* unexpected Out Of Memory (OOM)
@ 2013-08-07 0:02 Olivier Bonvalet
2013-08-07 8:29 ` [Xen-users] " Ian Campbell
0 siblings, 1 reply; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 0:02 UTC (permalink / raw)
To: xen-users; +Cc: xen-devel
Hi,
I have a memory problem on several PV domU, I *suppose* since the
upgrade to 3.10.4 kernel (and still occur with 3.10.5).
I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy
with a vanilla 3.9.11 kernel.
The problem : once a domU throw oom-killer, then it loop until all
processes will be killed, then the kernel abort and shutdown the VM.
I'm not sure how to read the OOM-killer messages about memory state, so
I copy/paste a list of them here, about a 2GB RAM domU :
[56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2
[56313.921413] 0000000000000000 ffffffff8134ab12 ffff880079cd2120 0000000000000000
[56313.921424] 00000000000280da ffffffff81089ad6 ffffffff8134f09a ffff88007773bba0
[56313.921433] ffffffff814d3d48 0000000000000000 00000000000280da 000000000000000b
[56313.921443] Call Trace:
[56313.921454] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
[56313.921462] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
[56313.921470] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[56313.921480] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
[56313.921486] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
[56313.921493] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
[56313.921502] [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc
[56313.921510] [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21
[56313.921516] [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed
[56313.921538] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
[56313.921547] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
[56313.921560] [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14
[56313.921566] [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133
[56313.921572] [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14
[56313.921579] [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3
[56313.921587] [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b
[56313.921593] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
[56313.921600] [<ffffffff81101c75>] ? seq_read+0xb8/0x31a
[56313.921606] [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40
[56313.921612] [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a
[56313.921632] [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74
[56313.921644] [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0
[56313.921650] [<ffffffff810e9e38>] ? SyS_read+0x41/0x70
[56313.921656] [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b
[56313.921662] Mem-Info:
[56313.921671] Node 0 DMA per-cpu:
[56313.921676] CPU 0: hi: 0, btch: 1 usd: 0
[56313.921680] Node 0 DMA32 per-cpu:
[56313.921684] CPU 0: hi: 186, btch: 31 usd: 83
[56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1
[56313.921693] active_file:153 inactive_file:849 isolated_file:0
[56313.921693] unevictable:0 dirty:0 writeback:0 unstable:0
[56313.921693] free:3392 slab_reclaimable:101577 slab_unreclaimable:232545
[56313.921693] mapped:3613 shmem:8327 pagetables:2848 bounce:0
[56313.921693] free_cma:0
[56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[56313.921726] lowmem_reserve[]: 0 1973 1973 1973
[56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB active_anon:90052kB inactive_anon:4132kB active_file:612kB inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:14452kB shmem:33308kB slab_reclaimable:404740kB slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 all_unreclaimable? yes
[56313.921749] lowmem_reserve[]: 0 0 0 0
[56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
[56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB
[56313.921791] 9341 total pagecache pages
[56313.921794] 0 pages in swap cache
[56313.921797] Swap cache stats: add 0, delete 0, find 0/0
[56313.921799] Free swap = 0kB
[56313.921802] Total swap = 0kB
[56313.927968] 524287 pages RAM
[56313.927976] 12090 pages reserved
[56313.927978] 307132 pages shared
[56313.927980] 430905 pages non-shared
[56313.927983] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[56313.927996] [ 330] 0 330 4213 114 12 0 -1000 udevd
[56313.928001] [ 815] 0 815 13018 114 20 0 0 rsyslogd
[56313.928006] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
[56313.928011] [ 909] 0 909 4212 113 12 0 -1000 udevd
[56313.928015] [ 910] 0 910 4212 113 12 0 -1000 udevd
[56313.928020] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
[56313.928025] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
[56313.928029] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
[56313.928034] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
[56313.928038] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
[56313.928043] [ 1016] 0 1016 86937 2033 155 0 0 apache2
[56313.928048] [ 1026] 0 1026 4692 40 14 0 0 atd
[56313.928057] [ 1054] 0 1054 5747 69 16 0 0 cron
[56313.928062] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
[56313.928067] [ 1104] 106 1104 9630 147 23 0 0 ntpd
[56313.928072] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
[56313.928077] [ 1130] 0 1130 18989 374 35 0 0 searchd
[56313.928081] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
[56313.928086] [ 1152] 105 1152 10265 1088 24 0 0 unbound
[56313.928090] [ 1172] 0 1172 1496 25 9 0 0 getty
[56313.928150] [ 1173] 0 1173 1496 24 8 0 0 getty
[56313.928157] [31557] 33 31557 88162 5277 161 0 0 apache2
[56313.928163] [31998] 33 31998 88044 5689 162 0 0 apache2
[56313.928168] [ 1031] 33 1031 87879 4726 161 0 0 apache2
[56313.928172] [ 1579] 33 1579 87493 5027 158 0 0 apache2
[56313.928177] [ 2901] 33 2901 87622 4707 158 0 0 apache2
[56313.928182] [ 3447] 33 3447 82447 4550 150 0 0 apache2
[56313.928186] [ 4559] 33 4559 87714 4945 160 0 0 apache2
[56313.928191] [ 4560] 33 4560 82914 4906 150 0 0 apache2
[56313.928195] [ 4561] 33 4561 87859 3840 159 0 0 apache2
[56313.928200] [ 6205] 33 6205 87422 3053 154 0 0 apache2
[56313.928204] [ 6776] 0 6776 10460 89 25 0 0 cron
[56313.928229] [ 6780] 0 6780 1002 23 8 0 0 sh
[56313.928236] [ 6784] 0 6784 7322 109 20 0 0 wget
[56313.928241] [ 6847] 0 6847 10460 89 25 0 0 cron
[56313.928246] [ 6856] 0 6856 1002 23 8 0 0 sh
[56313.928253] [ 6862] 0 6862 7322 108 20 0 0 wget
[56313.928258] [ 6924] 0 6924 10460 89 25 0 0 cron
[56313.928263] [ 6928] 0 6928 1002 24 8 0 0 sh
[56313.928267] [ 6932] 0 6932 7322 107 20 0 0 wget
[56313.928272] [ 6948] 33 6948 82020 2026 135 0 0 apache2
[56313.928277] [ 6991] 33 6991 86937 1966 137 0 0 apache2
[56313.928281] [ 6992] 33 6992 86937 1966 137 0 0 apache2
[56313.928285] [ 6993] 0 6993 10460 106 25 0 0 cron
[56313.928290] [ 6994] 0 6994 10460 106 25 0 0 cron
[56313.928294] [ 6995] 0 6995 10460 106 25 0 0 cron
[56313.928298] [ 6999] 1337 6999 10459 183 23 0 0 cron
[56313.928302] [ 7000] 0 7000 10459 182 23 0 0 cron
[56313.928306] [ 7001] 0 7001 10460 182 23 0 0 cron
[56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or sacrifice child
[56313.928315] Killed process 31998 (apache2) total-vm:352176kB, anon-rss:10848kB, file-rss:11908kB
[56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, oom_score_adj=0
[56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2
[56317.522115] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff8800793078e8
[56317.522125] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001802120
[56317.522133] ffffffff814d3d48 0000000000000000 00000000002840d0 000000000000000a
[56317.522142] Call Trace:
[56317.522153] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
[56317.522161] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
[56317.522168] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[56317.522177] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
[56317.522184] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
[56317.522190] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
[56317.522197] [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5
[56317.522203] [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b
[56317.522211] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
[56317.522219] [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d
[56317.522226] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
[56317.522232] [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f
[56317.522238] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
[56317.522245] [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92
[56317.522251] [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31
[56317.522258] [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105
[56317.522264] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
[56317.522271] [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab
[56317.522277] [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b
[56317.522283] [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f
[56317.522289] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
[56317.522296] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
[56317.522304] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
[56317.522310] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
[56317.522440] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
[56317.522449] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
[56317.522457] [<ffffffff81113891>] ? fsnotify+0x261/0x291
[56317.522464] [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20
[56317.522471] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
[56317.522479] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
[56317.522498] [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72
[56317.522503] [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3
[56317.522508] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
[56317.522512] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
[56317.522517] [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55
[56317.522522] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
[56317.522526] Mem-Info:
[56317.522529] Node 0 DMA per-cpu:
[56317.522540] CPU 0: hi: 0, btch: 1 usd: 0
[56317.522544] Node 0 DMA32 per-cpu:
[56317.522549] CPU 0: hi: 186, btch: 31 usd: 90
[56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0
[56317.522556] active_file:95 inactive_file:793 isolated_file:0
[56317.522556] unevictable:0 dirty:0 writeback:0 unstable:0
[56317.522556] free:3378 slab_reclaimable:101712 slab_unreclaimable:232523
[56317.522556] mapped:3140 shmem:8327 pagetables:2478 bounce:0
[56317.522556] free_cma:0
[56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[56317.522590] lowmem_reserve[]: 0 1973 1973 1973
[56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB active_anon:91128kB inactive_anon:4132kB active_file:380kB inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12560kB shmem:33308kB slab_reclaimable:405280kB slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 all_unreclaimable? yes
[56317.522621] lowmem_reserve[]: 0 0 0 0
[56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
[56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB
[56317.522660] 9223 total pagecache pages
[56317.522663] 0 pages in swap cache
[56317.522668] Swap cache stats: add 0, delete 0, find 0/0
[56317.522671] Free swap = 0kB
[56317.522673] Total swap = 0kB
[56317.529054] 524287 pages RAM
[56317.529064] 12090 pages reserved
[56317.529068] 294744 pages shared
[56317.529071] 431526 pages non-shared
[56317.529075] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[56317.529090] [ 330] 0 330 4213 114 12 0 -1000 udevd
[56317.529098] [ 815] 0 815 13018 110 20 0 0 rsyslogd
[56317.529105] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
[56317.529112] [ 909] 0 909 4212 113 12 0 -1000 udevd
[56317.529119] [ 910] 0 910 4212 113 12 0 -1000 udevd
[56317.529126] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
[56317.529133] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
[56317.529139] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
[56317.529146] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
[56317.529152] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
[56317.529159] [ 1016] 0 1016 86937 2043 155 0 0 apache2
[56317.529166] [ 1026] 0 1026 4692 40 14 0 0 atd
[56317.529177] [ 1054] 0 1054 5747 69 16 0 0 cron
[56317.529184] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
[56317.529190] [ 1104] 106 1104 9630 153 23 0 0 ntpd
[56317.529197] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
[56317.529203] [ 1130] 0 1130 18989 395 35 0 0 searchd
[56317.529211] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
[56317.529220] [ 1152] 105 1152 10265 1088 24 0 0 unbound
[56317.529228] [ 1172] 0 1172 1496 25 9 0 0 getty
[56317.529237] [ 1173] 0 1173 1496 24 8 0 0 getty
[56317.529245] [31557] 33 31557 88162 5277 161 0 0 apache2
[56317.529252] [ 1031] 33 1031 87879 4726 161 0 0 apache2
[56317.529261] [ 1579] 33 1579 87493 5027 158 0 0 apache2
[56317.529267] [ 2901] 33 2901 87622 4707 158 0 0 apache2
[56317.529274] [ 3447] 33 3447 35706 2055 71 0 0 apache2
[56317.529281] [ 4559] 33 4559 87714 4945 160 0 0 apache2
[56317.529287] [ 4560] 33 4560 35648 1981 71 0 0 apache2
[56317.529296] [ 4561] 33 4561 87859 3840 159 0 0 apache2
[56317.529304] [ 6205] 33 6205 87422 3053 154 0 0 apache2
[56317.529311] [ 6776] 0 6776 10460 89 25 0 0 cron
[56317.529317] [ 6780] 0 6780 1002 23 8 0 0 sh
[56317.529324] [ 6784] 0 6784 7322 109 20 0 0 wget
[56317.529331] [ 6847] 0 6847 10460 89 25 0 0 cron
[56317.529338] [ 6856] 0 6856 1002 23 8 0 0 sh
[56317.529345] [ 6862] 0 6862 7322 108 20 0 0 wget
[56317.529351] [ 6924] 0 6924 10460 89 25 0 0 cron
[56317.529361] [ 6928] 0 6928 1002 24 8 0 0 sh
[56317.529367] [ 6932] 0 6932 7322 107 20 0 0 wget
[56317.529374] [ 6948] 33 6948 35279 1532 70 0 0 apache2
[56317.529381] [ 6991] 33 6991 86937 1966 137 0 0 apache2
[56317.529388] [ 6992] 33 6992 86937 1966 137 0 0 apache2
[56317.529394] [ 6993] 0 6993 10460 88 25 0 0 cron
[56317.529403] [ 6994] 0 6994 10460 88 25 0 0 cron
[56317.529410] [ 6995] 0 6995 10460 88 25 0 0 cron
[56317.529416] [ 6999] 1337 6999 1002 23 8 0 0 sh
[56317.529423] [ 7000] 0 7000 1002 24 8 0 0 sh
[56317.529429] [ 7001] 0 7001 1002 23 8 0 0 sh
[56317.529436] [ 7002] 1337 7002 1481 21 9 0 0 sleep
[56317.529445] [ 7003] 0 7003 7322 110 20 0 0 wget
[56317.529452] [ 7004] 0 7004 7322 110 20 0 0 wget
[56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or sacrifice child
[56317.529464] Killed process 31557 (apache2) total-vm:352648kB, anon-rss:11004kB, file-rss:10104kB
[56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2
[56680.628607] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006e3ffbb0
[56680.628636] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001a8c8e0
[56680.628645] ffffffff814d3d48 0000000000000000 00000000000201da 000000000000000b
[56680.628654] Call Trace:
[56680.628666] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
[56680.628675] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
[56680.628682] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[56680.628692] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
[56680.628699] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
[56680.628706] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
[56680.628713] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
[56680.628720] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
[56680.628727] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
[56680.628735] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
[56680.628741] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
[56680.628748] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
[56680.628756] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
[56680.628763] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
[56680.628768] Mem-Info:
[56680.628771] Node 0 DMA per-cpu:
[56680.628776] CPU 0: hi: 0, btch: 1 usd: 0
[56680.628782] Node 0 DMA32 per-cpu:
[56680.628788] CPU 0: hi: 186, btch: 31 usd: 115
[56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0
[56680.628796] active_file:156 inactive_file:1004 isolated_file:0
[56680.628796] unevictable:0 dirty:0 writeback:0 unstable:0
[56680.628796] free:3382 slab_reclaimable:103079 slab_unreclaimable:233653
[56680.628796] mapped:3205 shmem:8327 pagetables:2067 bounce:0
[56680.628796] free_cma:0
[56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[56680.628848] lowmem_reserve[]: 0 1973 1973 1973
[56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB active_anon:78780kB inactive_anon:4132kB active_file:624kB inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12820kB shmem:33308kB slab_reclaimable:410684kB slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 all_unreclaimable? yes
[56680.628883] lowmem_reserve[]: 0 0 0 0
[56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
[56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB
[56680.628945] 9494 total pagecache pages
[56680.628949] 0 pages in swap cache
[56680.628957] Swap cache stats: add 0, delete 0, find 0/0
[56680.628962] Free swap = 0kB
[56680.628965] Total swap = 0kB
[56680.634908] 524287 pages RAM
[56680.634916] 12090 pages reserved
[56680.634919] 291027 pages shared
[56680.634921] 431125 pages non-shared
[56680.634924] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[56680.634937] [ 330] 0 330 4213 114 12 0 -1000 udevd
[56680.634943] [ 815] 0 815 13018 138 20 0 0 rsyslogd
[56680.634948] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
[56680.634953] [ 909] 0 909 4212 113 12 0 -1000 udevd
[56680.634957] [ 910] 0 910 4212 113 12 0 -1000 udevd
[56680.634961] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
[56680.634966] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
[56680.634971] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
[56680.634975] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
[56680.634980] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
[56680.634985] [ 1016] 0 1016 86937 2014 155 0 0 apache2
[56680.634990] [ 1026] 0 1026 4692 40 14 0 0 atd
[56680.634995] [ 1054] 0 1054 5747 69 16 0 0 cron
[56680.634999] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
[56680.635004] [ 1104] 106 1104 9630 147 23 0 0 ntpd
[56680.635008] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
[56680.635013] [ 1130] 0 1130 18989 400 35 0 0 searchd
[56680.635018] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
[56680.635022] [ 1152] 105 1152 10265 1099 24 0 0 unbound
[56680.635027] [ 1172] 0 1172 1496 25 9 0 0 getty
[56680.635144] [ 1173] 0 1173 1496 24 8 0 0 getty
[56680.635150] [ 1031] 33 1031 87879 4814 161 0 0 apache2
[56680.635155] [ 1579] 33 1579 87941 5548 159 0 0 apache2
[56680.635160] [ 2901] 33 2901 87622 4692 158 0 0 apache2
[56680.635164] [ 4559] 33 4559 87714 4947 160 0 0 apache2
[56680.635169] [ 4561] 33 4561 87859 3852 159 0 0 apache2
[56680.635174] [ 6205] 33 6205 87294 3684 155 0 0 apache2
[56680.635179] [ 6991] 33 6991 87434 3395 152 0 0 apache2
[56680.635183] [ 6992] 33 6992 87593 3185 148 0 0 apache2
[56680.635188] [ 7289] 0 7289 10460 89 25 0 0 cron
[56680.635192] [ 7297] 0 7297 1002 23 7 0 0 sh
[56680.635197] [ 7303] 0 7303 7322 110 20 0 0 wget
[56680.635202] [ 7363] 0 7363 10460 89 25 0 0 cron
[56680.635206] [ 7374] 0 7374 1002 25 7 0 0 sh
[56680.635213] [ 7378] 0 7378 7322 108 19 0 0 wget
[56680.635218] [ 7437] 0 7437 10460 89 25 0 0 cron
[56680.635222] [ 7438] 0 7438 10460 89 25 0 0 cron
[56680.635227] [ 7439] 0 7439 10460 89 25 0 0 cron
[56680.635231] [ 7443] 0 7443 1002 23 8 0 0 sh
[56680.635235] [ 7444] 0 7444 1002 23 8 0 0 sh
[56680.635239] [ 7445] 1337 7445 1002 24 7 0 0 sh
[56680.635244] [ 7446] 1337 7446 1481 22 8 0 0 sleep
[56680.635248] [ 7447] 0 7447 7322 109 19 0 0 wget
[56680.635252] [ 7448] 0 7448 7322 107 19 0 0 wget
[56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or sacrifice child
[56680.635262] Killed process 1579 (apache2) total-vm:351764kB, anon-rss:12092kB, file-rss:10100kB
[56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2
[56734.778963] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006d133bb0
[56734.778973] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800634686a0
[56734.778982] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
[56734.779011] Call Trace:
[56734.779020] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
[56734.779026] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
[56734.779031] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[56734.779037] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
[56734.779042] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
[56734.779046] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
[56734.779051] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
[56734.779055] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
[56734.779060] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
[56734.779065] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
[56734.779069] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
[56734.779073] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
[56734.779079] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
[56734.779085] [<ffffffff81113891>] ? fsnotify+0x261/0x291
[56734.779089] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
[56734.779093] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
[56734.779097] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
[56734.779100] Mem-Info:
[56734.779102] Node 0 DMA per-cpu:
[56734.779106] CPU 0: hi: 0, btch: 1 usd: 0
[56734.779108] Node 0 DMA32 per-cpu:
[56734.779112] CPU 0: hi: 186, btch: 31 usd: 62
[56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0
[56734.779117] active_file:190 inactive_file:1109 isolated_file:0
[56734.779117] unevictable:0 dirty:2 writeback:0 unstable:0
[56734.779117] free:3395 slab_reclaimable:103429 slab_unreclaimable:233857
[56734.779117] mapped:3186 shmem:8327 pagetables:2038 bounce:0
[56734.779117] free_cma:0
[56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[56734.779256] lowmem_reserve[]: 0 1973 1973 1973
[56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:75512kB inactive_anon:4132kB active_file:760kB inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB mapped:12744kB shmem:33308kB slab_reclaimable:412052kB slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 all_unreclaimable? yes
[56734.779293] lowmem_reserve[]: 0 0 0 0
[56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
[56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
[56734.779351] 9645 total pagecache pages
[56734.779359] 0 pages in swap cache
[56734.779363] Swap cache stats: add 0, delete 0, find 0/0
[56734.779367] Free swap = 0kB
[56734.779370] Total swap = 0kB
[56734.785510] 524287 pages RAM
[56734.785519] 12090 pages reserved
[56734.785523] 286620 pages shared
[56734.785526] 431108 pages non-shared
[56734.785529] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[56734.785544] [ 330] 0 330 4213 114 12 0 -1000 udevd
[56734.785553] [ 815] 0 815 13018 138 20 0 0 rsyslogd
[56734.785560] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
[56734.785567] [ 909] 0 909 4212 113 12 0 -1000 udevd
[56734.785573] [ 910] 0 910 4212 113 12 0 -1000 udevd
[56734.785579] [ 911] 104 911 12487 208 27 0 0 zabbix_agentd
[56734.785586] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
[56734.785593] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
[56734.785600] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
[56734.785607] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
[56734.785614] [ 1016] 0 1016 86937 2024 155 0 0 apache2
[56734.785620] [ 1026] 0 1026 4692 40 14 0 0 atd
[56734.785627] [ 1054] 0 1054 5747 70 16 0 0 cron
[56734.785634] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
[56734.785641] [ 1104] 106 1104 9630 147 23 0 0 ntpd
[56734.785648] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
[56734.785654] [ 1130] 0 1130 18989 395 35 0 0 searchd
[56734.785661] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
[56734.785665] [ 1152] 105 1152 10265 1099 24 0 0 unbound
[56734.785670] [ 1172] 0 1172 1496 25 9 0 0 getty
[56734.785712] [ 1173] 0 1173 1496 24 8 0 0 getty
[56734.785718] [ 1031] 33 1031 87879 4814 161 0 0 apache2
[56734.785722] [ 2901] 33 2901 87622 4692 158 0 0 apache2
[56734.785727] [ 4559] 33 4559 87714 4954 160 0 0 apache2
[56734.785731] [ 4561] 33 4561 87859 3815 159 0 0 apache2
[56734.785737] [ 6205] 33 6205 87294 3674 155 0 0 apache2
[56734.785741] [ 6991] 33 6991 87434 3398 152 0 0 apache2
[56734.785745] [ 6992] 33 6992 87593 3228 150 0 0 apache2
[56734.785750] [ 7363] 0 7363 10460 89 25 0 0 cron
[56734.785754] [ 7374] 0 7374 1002 25 7 0 0 sh
[56734.785760] [ 7378] 0 7378 7322 108 19 0 0 wget
[56734.785765] [ 7505] 0 7505 10460 89 25 0 0 cron
[56734.785769] [ 7506] 0 7506 10460 89 25 0 0 cron
[56734.785773] [ 7507] 0 7507 10460 89 25 0 0 cron
[56734.785777] [ 7508] 0 7508 10460 89 25 0 0 cron
[56734.785782] [ 7509] 0 7509 10460 89 25 0 0 cron
[56734.785786] [ 7510] 0 7510 10460 89 25 0 0 cron
[56734.785790] [ 7517] 0 7517 1002 24 8 0 0 sh
[56734.785794] [ 7518] 0 7518 1002 24 8 0 0 sh
[56734.785800] [ 7519] 0 7519 1002 24 8 0 0 sh
[56734.785804] [ 7520] 0 7520 1002 23 8 0 0 sh
[56734.785808] [ 7521] 1337 7521 1002 24 8 0 0 sh
[56734.785813] [ 7522] 0 7522 1002 23 7 0 0 sh
[56734.785817] [ 7523] 1337 7523 1481 21 8 0 0 sleep
[56734.785822] [ 7524] 0 7524 7322 148 20 0 0 wget
[56734.785826] [ 7525] 0 7525 7322 111 20 0 0 wget
[56734.785830] [ 7526] 0 7526 7322 110 19 0 0 wget
[56734.785834] [ 7527] 0 7527 7322 110 20 0 0 wget
[56734.785839] [ 7528] 0 7528 1002 24 7 0 0 maxlifetime
[56734.785846] [ 7530] 0 7530 16356 291 37 0 0 php5
[56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or sacrifice child
[56734.785854] Killed process 4559 (apache2) total-vm:350856kB, anon-rss:11596kB, file-rss:8220kB
[56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2
[56793.525928] 0000000000000000 ffffffff8134ab12 ffffffff8134f478 00000000000007c5
[56793.525938] 0000000000000010 ffffffff81089ad6 ffffffff8134f09a ffff880001a127c0
[56793.525946] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
[56793.525976] Call Trace:
[56793.525986] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
[56793.526012] [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6
[56793.526042] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
[56793.526048] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[56793.526057] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
[56793.526064] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
[56793.526070] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
[56793.526082] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
[56793.526090] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
[56793.526097] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
[56793.526104] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
[56793.526111] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
[56793.526117] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
[56793.526125] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
[56793.526133] [<ffffffff81113891>] ? fsnotify+0x261/0x291
[56793.526142] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
[56793.526149] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
[56793.526155] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
[56793.526160] Mem-Info:
[56793.526163] Node 0 DMA per-cpu:
[56793.526171] CPU 0: hi: 0, btch: 1 usd: 0
[56793.526175] Node 0 DMA32 per-cpu:
[56793.526180] CPU 0: hi: 186, btch: 31 usd: 150
[56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5
[56793.526188] active_file:78 inactive_file:778 isolated_file:0
[56793.526188] unevictable:0 dirty:0 writeback:0 unstable:0
[56793.526188] free:3395 slab_reclaimable:104385 slab_unreclaimable:234096
[56793.526188] mapped:2974 shmem:8327 pagetables:2047 bounce:0
[56793.526188] free_cma:0
[56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes
[56793.526238] lowmem_reserve[]: 0 1973 1973 1973
[56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:71364kB inactive_anon:4132kB active_file:312kB inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:11896kB shmem:33308kB slab_reclaimable:415876kB slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 all_unreclaimable? yes
[56793.526277] lowmem_reserve[]: 0 0 0 0
[56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
[56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
[56793.526343] 9197 total pagecache pages
[56793.526347] 0 pages in swap cache
[56793.526352] Swap cache stats: add 0, delete 0, find 0/0
[56793.526356] Free swap = 0kB
[56793.526359] Total swap = 0kB
[56793.532348] 524287 pages RAM
[56793.532355] 12090 pages reserved
[56793.532358] 284892 pages shared
[56793.532360] 430988 pages non-shared
[56793.532363] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[56793.532376] [ 330] 0 330 4213 114 12 0 -1000 udevd
[56793.532381] [ 815] 0 815 13018 138 20 0 0 rsyslogd
[56793.532386] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
[56793.532391] [ 909] 0 909 4212 113 12 0 -1000 udevd
[56793.532396] [ 910] 0 910 4212 113 12 0 -1000 udevd
[56793.532400] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
[56793.532405] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
[56793.532409] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
[56793.532414] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
[56793.532418] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
[56793.532424] [ 1016] 0 1016 86937 2028 155 0 0 apache2
[56793.532428] [ 1026] 0 1026 4692 40 14 0 0 atd
[56793.532433] [ 1054] 0 1054 5747 70 16 0 0 cron
[56793.532437] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
[56793.532442] [ 1104] 106 1104 9630 147 23 0 0 ntpd
[56793.532447] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
[56793.532451] [ 1130] 0 1130 18989 374 35 0 0 searchd
[56793.532456] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
[56793.532461] [ 1152] 105 1152 10265 1099 24 0 0 unbound
[56793.532467] [ 1172] 0 1172 1496 25 9 0 0 getty
[56793.532484] [ 1173] 0 1173 1496 24 8 0 0 getty
[56793.532491] [ 1031] 33 1031 87879 4791 161 0 0 apache2
[56793.532497] [ 2901] 33 2901 87622 4723 158 0 0 apache2
[56793.532504] [ 4561] 33 4561 87859 3866 159 0 0 apache2
[56793.532510] [ 6205] 33 6205 87294 3663 155 0 0 apache2
[56793.532515] [ 6991] 33 6991 87434 3398 152 0 0 apache2
[56793.532522] [ 6992] 33 6992 87593 3228 150 0 0 apache2
[56793.532526] [ 7363] 0 7363 10460 89 25 0 0 cron
[56793.532531] [ 7374] 0 7374 1002 25 7 0 0 sh
[56793.532537] [ 7378] 0 7378 7322 108 19 0 0 wget
[56793.532547] [ 7555] 33 7555 86937 1967 137 0 0 apache2
[56793.532553] [ 7593] 0 7593 10460 88 25 0 0 cron
[56793.532559] [ 7594] 0 7594 10460 88 25 0 0 cron
[56793.532564] [ 7595] 0 7595 10460 88 25 0 0 cron
[56793.532568] [ 7596] 0 7596 10460 88 25 0 0 cron
[56793.532573] [ 7597] 0 7597 10460 88 25 0 0 cron
[56793.532580] [ 7598] 0 7598 10460 88 25 0 0 cron
[56793.532584] [ 7599] 0 7599 10460 88 25 0 0 cron
[56793.532588] [ 7607] 0 7607 1002 25 8 0 0 sh
[56793.532593] [ 7608] 1337 7608 1002 24 8 0 0 sh
[56793.532597] [ 7609] 0 7609 1002 26 7 0 0 sh
[56793.532601] [ 7610] 0 7610 1002 24 7 0 0 sh
[56793.532606] [ 7611] 0 7611 1002 25 7 0 0 sh
[56793.532611] [ 7612] 0 7612 1002 23 8 0 0 sh
[56793.532615] [ 7613] 0 7613 1002 25 7 0 0 sh
[56793.532620] [ 7614] 0 7614 7322 156 20 0 0 wget
[56793.532624] [ 7615] 1337 7615 1481 19 8 0 0 sleep
[56793.532628] [ 7616] 0 7616 7322 110 19 0 0 wget
[56793.532632] [ 7617] 0 7617 7322 110 20 0 0 wget
[56793.532637] [ 7618] 0 7618 7322 109 19 0 0 wget
[56793.532641] [ 7619] 0 7619 7322 109 20 0 0 wget
[56793.532646] [ 7620] 0 7620 7322 109 20 0 0 wget
[56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or sacrifice child
[56793.532656] Killed process 1031 (apache2) total-vm:351516kB, anon-rss:9624kB, file-rss:9540kB
...
[57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted 2.6.50-dae-xen #2
[57131.763847] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff880073e21bb0
[57131.763858] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800788aea00
[57131.763867] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000004
[57131.763885] Call Trace:
[57131.763896] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
[57131.763906] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
[57131.763913] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[57131.763921] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
[57131.763947] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
[57131.763955] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
[57131.763962] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
[57131.763969] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
[57131.763977] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
[57131.763985] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
[57131.763991] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
[57131.763997] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
[57131.764006] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
[57131.764017] [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182
[57131.764023] [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7
[57131.764030] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
[57131.764034] Mem-Info:
[57131.764037] Node 0 DMA per-cpu:
[57131.764041] CPU 0: hi: 0, btch: 1 usd: 0
[57131.764045] Node 0 DMA32 per-cpu:
[57131.764049] CPU 0: hi: 186, btch: 31 usd: 62
[57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0
[57131.764076] active_file:288 inactive_file:1121 isolated_file:0
[57131.764076] unevictable:0 dirty:0 writeback:0 unstable:0
[57131.764076] free:3369 slab_reclaimable:109088 slab_unreclaimable:235420
[57131.764076] mapped:358 shmem:8328 pagetables:821 bounce:0
[57131.764076] free_cma:0
[57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[57131.764115] lowmem_reserve[]: 0 1973 1973 1973
[57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB active_anon:46972kB inactive_anon:4104kB active_file:1152kB inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:1432kB shmem:33312kB slab_reclaimable:434592kB slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 all_unreclaimable? yes
[57131.764151] lowmem_reserve[]: 0 0 0 0
[57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
[57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5548kB
[57131.764242] 9744 total pagecache pages
[57131.764253] 0 pages in swap cache
[57131.764256] Swap cache stats: add 0, delete 0, find 0/0
[57131.764263] Free swap = 0kB
[57131.764267] Total swap = 0kB
[57131.770252] 524287 pages RAM
[57131.770260] 12090 pages reserved
[57131.770262] 265440 pages shared
[57131.770265] 433233 pages non-shared
[57131.770267] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[57131.770280] [ 330] 0 330 4213 114 12 0 -1000 udevd
[57131.770285] [ 815] 0 815 13048 150 20 0 0 rsyslogd
[57131.770290] [ 853] 104 853 12486 99 27 0 0 zabbix_agentd
[57131.770295] [ 909] 0 909 4212 113 12 0 -1000 udevd
[57131.770299] [ 910] 0 910 4212 113 12 0 -1000 udevd
[57131.770305] [ 1016] 0 1016 86937 2031 155 0 0 apache2
[57131.770310] [ 1054] 0 1054 5747 69 16 0 0 cron
[57131.770314] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
[57131.770319] [ 1104] 106 1104 9630 147 23 0 0 ntpd
[57131.770323] [ 1130] 0 1130 18989 374 35 0 0 searchd
[57131.770328] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
[57131.770334] [ 7363] 0 7363 10460 89 25 0 0 cron
[57131.770339] [ 7374] 0 7374 1002 25 7 0 0 sh
[57131.770343] [ 7378] 0 7378 7322 108 19 0 0 wget
[57131.770348] [ 8003] 0 8003 10460 88 25 0 0 cron
[57131.770352] [ 8006] 0 8006 10460 88 25 0 0 cron
[57131.770357] [ 8009] 0 8009 10460 88 25 0 0 cron
[57131.770361] [ 8020] 1337 8020 1002 23 7 0 0 sh
[57131.770366] [ 8023] 0 8023 1002 23 6 0 0 sh
[57131.770370] [ 8024] 0 8024 1002 24 7 0 0 sh
[57131.770399] [ 8030] 0 8030 7322 108 19 0 0 wget
[57131.770406] [ 8035] 0 8035 7322 109 20 0 0 wget
[57131.770410] [ 8051] 0 8051 1002 41 8 0 0 rc
[57131.770414] [ 8054] 0 8054 3085 60 11 0 0 startpar
[57131.770419] [ 8055] 1337 8055 1002 23 8 0 0 zabbix-launch-t
[57131.770424] [ 8057] 0 8057 1002 40 7 0 0 apache2
[57131.770430] [ 8067] 1337 8067 2841 76 9 0 0 apache
[57131.770436] [ 8084] 0 8084 1002 24 8 0 0 apache2ctl
[57131.770443] [ 8092] 0 8092 26561 288 48 0 0 apache2
[57131.770450] [ 8115] 0 8115 1002 22 8 0 0 zabbix-agent
[57131.770457] [ 8116] 0 8116 980 32 8 0 0 start-stop-daem
[57131.770461] [ 8158] 1337 8158 10130 168 22 0 0 zabbix_sender
[57131.770466] [ 8161] 33 8161 86937 1967 137 0 0 apache2
[57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or sacrifice child
[57131.770475] Killed process 8161 (apache2) total-vm:347748kB, anon-rss:7868kB, file-rss:0kB
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 0:02 unexpected Out Of Memory (OOM) Olivier Bonvalet
@ 2013-08-07 8:29 ` Ian Campbell
2013-08-07 8:58 ` Olivier Bonvalet
0 siblings, 1 reply; 18+ messages in thread
From: Ian Campbell @ 2013-08-07 8:29 UTC (permalink / raw)
To: Olivier Bonvalet; +Cc: xen-users, xen-devel
On Wed, 2013-08-07 at 02:02 +0200, Olivier Bonvalet wrote:
> Hi,
>
> I have a memory problem on several PV domU, I *suppose* since the
> upgrade to 3.10.4 kernel (and still occur with 3.10.5).
> I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy
> with a vanilla 3.9.11 kernel.
>
> The problem : once a domU throw oom-killer, then it loop until all
> processes will be killed, then the kernel abort and shutdown the VM.
>
> I'm not sure how to read the OOM-killer messages about memory state, so
> I copy/paste a list of them here, about a 2GB RAM domU :
Does /proc/meminfo on the VM show that it actually has 2GB?
> [56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
> [56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2
This appears to show a 2.6.50 kernel, whatever that is, and not 3.10.x?
> [56313.921413] 0000000000000000 ffffffff8134ab12 ffff880079cd2120 0000000000000000
> [56313.921424] 00000000000280da ffffffff81089ad6 ffffffff8134f09a ffff88007773bba0
> [56313.921433] ffffffff814d3d48 0000000000000000 00000000000280da 000000000000000b
> [56313.921443] Call Trace:
> [56313.921454] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56313.921462] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56313.921470] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56313.921480] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56313.921486] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56313.921493] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56313.921502] [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc
> [56313.921510] [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21
> [56313.921516] [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed
> [56313.921538] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56313.921547] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56313.921560] [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14
> [56313.921566] [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133
> [56313.921572] [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14
> [56313.921579] [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3
> [56313.921587] [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b
> [56313.921593] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56313.921600] [<ffffffff81101c75>] ? seq_read+0xb8/0x31a
> [56313.921606] [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40
> [56313.921612] [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a
> [56313.921632] [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74
> [56313.921644] [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0
> [56313.921650] [<ffffffff810e9e38>] ? SyS_read+0x41/0x70
> [56313.921656] [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b
> [56313.921662] Mem-Info:
> [56313.921671] Node 0 DMA per-cpu:
> [56313.921676] CPU 0: hi: 0, btch: 1 usd: 0
> [56313.921680] Node 0 DMA32 per-cpu:
> [56313.921684] CPU 0: hi: 186, btch: 31 usd: 83
> [56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1
> [56313.921693] active_file:153 inactive_file:849 isolated_file:0
> [56313.921693] unevictable:0 dirty:0 writeback:0 unstable:0
> [56313.921693] free:3392 slab_reclaimable:101577 slab_unreclaimable:232545
> [56313.921693] mapped:3613 shmem:8327 pagetables:2848 bounce:0
> [56313.921693] free_cma:0
> [56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> [56313.921726] lowmem_reserve[]: 0 1973 1973 1973
> [56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB active_anon:90052kB inactive_anon:4132kB active_file:612kB inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:14452kB shmem:33308kB slab_reclaimable:404740kB slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 all_unreclaimable? yes
> [56313.921749] lowmem_reserve[]: 0 0 0 0
> [56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
> [56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB
> [56313.921791] 9341 total pagecache pages
> [56313.921794] 0 pages in swap cache
> [56313.921797] Swap cache stats: add 0, delete 0, find 0/0
> [56313.921799] Free swap = 0kB
> [56313.921802] Total swap = 0kB
> [56313.927968] 524287 pages RAM
> [56313.927976] 12090 pages reserved
> [56313.927978] 307132 pages shared
> [56313.927980] 430905 pages non-shared
> [56313.927983] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> [56313.927996] [ 330] 0 330 4213 114 12 0 -1000 udevd
> [56313.928001] [ 815] 0 815 13018 114 20 0 0 rsyslogd
> [56313.928006] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> [56313.928011] [ 909] 0 909 4212 113 12 0 -1000 udevd
> [56313.928015] [ 910] 0 910 4212 113 12 0 -1000 udevd
> [56313.928020] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
> [56313.928025] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> [56313.928029] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> [56313.928034] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> [56313.928038] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> [56313.928043] [ 1016] 0 1016 86937 2033 155 0 0 apache2
> [56313.928048] [ 1026] 0 1026 4692 40 14 0 0 atd
> [56313.928057] [ 1054] 0 1054 5747 69 16 0 0 cron
> [56313.928062] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> [56313.928067] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> [56313.928072] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> [56313.928077] [ 1130] 0 1130 18989 374 35 0 0 searchd
> [56313.928081] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> [56313.928086] [ 1152] 105 1152 10265 1088 24 0 0 unbound
> [56313.928090] [ 1172] 0 1172 1496 25 9 0 0 getty
> [56313.928150] [ 1173] 0 1173 1496 24 8 0 0 getty
> [56313.928157] [31557] 33 31557 88162 5277 161 0 0 apache2
> [56313.928163] [31998] 33 31998 88044 5689 162 0 0 apache2
> [56313.928168] [ 1031] 33 1031 87879 4726 161 0 0 apache2
> [56313.928172] [ 1579] 33 1579 87493 5027 158 0 0 apache2
> [56313.928177] [ 2901] 33 2901 87622 4707 158 0 0 apache2
> [56313.928182] [ 3447] 33 3447 82447 4550 150 0 0 apache2
> [56313.928186] [ 4559] 33 4559 87714 4945 160 0 0 apache2
> [56313.928191] [ 4560] 33 4560 82914 4906 150 0 0 apache2
> [56313.928195] [ 4561] 33 4561 87859 3840 159 0 0 apache2
> [56313.928200] [ 6205] 33 6205 87422 3053 154 0 0 apache2
> [56313.928204] [ 6776] 0 6776 10460 89 25 0 0 cron
> [56313.928229] [ 6780] 0 6780 1002 23 8 0 0 sh
> [56313.928236] [ 6784] 0 6784 7322 109 20 0 0 wget
> [56313.928241] [ 6847] 0 6847 10460 89 25 0 0 cron
> [56313.928246] [ 6856] 0 6856 1002 23 8 0 0 sh
> [56313.928253] [ 6862] 0 6862 7322 108 20 0 0 wget
> [56313.928258] [ 6924] 0 6924 10460 89 25 0 0 cron
> [56313.928263] [ 6928] 0 6928 1002 24 8 0 0 sh
> [56313.928267] [ 6932] 0 6932 7322 107 20 0 0 wget
> [56313.928272] [ 6948] 33 6948 82020 2026 135 0 0 apache2
> [56313.928277] [ 6991] 33 6991 86937 1966 137 0 0 apache2
> [56313.928281] [ 6992] 33 6992 86937 1966 137 0 0 apache2
> [56313.928285] [ 6993] 0 6993 10460 106 25 0 0 cron
> [56313.928290] [ 6994] 0 6994 10460 106 25 0 0 cron
> [56313.928294] [ 6995] 0 6995 10460 106 25 0 0 cron
> [56313.928298] [ 6999] 1337 6999 10459 183 23 0 0 cron
> [56313.928302] [ 7000] 0 7000 10459 182 23 0 0 cron
> [56313.928306] [ 7001] 0 7001 10460 182 23 0 0 cron
> [56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or sacrifice child
> [56313.928315] Killed process 31998 (apache2) total-vm:352176kB, anon-rss:10848kB, file-rss:11908kB
> [56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, oom_score_adj=0
> [56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2
> [56317.522115] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff8800793078e8
> [56317.522125] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001802120
> [56317.522133] ffffffff814d3d48 0000000000000000 00000000002840d0 000000000000000a
> [56317.522142] Call Trace:
> [56317.522153] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56317.522161] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56317.522168] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56317.522177] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56317.522184] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56317.522190] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56317.522197] [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5
> [56317.522203] [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b
> [56317.522211] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> [56317.522219] [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d
> [56317.522226] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> [56317.522232] [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f
> [56317.522238] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> [56317.522245] [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92
> [56317.522251] [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31
> [56317.522258] [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105
> [56317.522264] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> [56317.522271] [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab
> [56317.522277] [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b
> [56317.522283] [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f
> [56317.522289] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56317.522296] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56317.522304] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56317.522310] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56317.522440] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56317.522449] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56317.522457] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> [56317.522464] [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20
> [56317.522471] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> [56317.522479] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
> [56317.522498] [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72
> [56317.522503] [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3
> [56317.522508] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> [56317.522512] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
> [56317.522517] [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55
> [56317.522522] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56317.522526] Mem-Info:
> [56317.522529] Node 0 DMA per-cpu:
> [56317.522540] CPU 0: hi: 0, btch: 1 usd: 0
> [56317.522544] Node 0 DMA32 per-cpu:
> [56317.522549] CPU 0: hi: 186, btch: 31 usd: 90
> [56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0
> [56317.522556] active_file:95 inactive_file:793 isolated_file:0
> [56317.522556] unevictable:0 dirty:0 writeback:0 unstable:0
> [56317.522556] free:3378 slab_reclaimable:101712 slab_unreclaimable:232523
> [56317.522556] mapped:3140 shmem:8327 pagetables:2478 bounce:0
> [56317.522556] free_cma:0
> [56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> [56317.522590] lowmem_reserve[]: 0 1973 1973 1973
> [56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB active_anon:91128kB inactive_anon:4132kB active_file:380kB inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12560kB shmem:33308kB slab_reclaimable:405280kB slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 all_unreclaimable? yes
> [56317.522621] lowmem_reserve[]: 0 0 0 0
> [56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
> [56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB
> [56317.522660] 9223 total pagecache pages
> [56317.522663] 0 pages in swap cache
> [56317.522668] Swap cache stats: add 0, delete 0, find 0/0
> [56317.522671] Free swap = 0kB
> [56317.522673] Total swap = 0kB
> [56317.529054] 524287 pages RAM
> [56317.529064] 12090 pages reserved
> [56317.529068] 294744 pages shared
> [56317.529071] 431526 pages non-shared
> [56317.529075] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> [56317.529090] [ 330] 0 330 4213 114 12 0 -1000 udevd
> [56317.529098] [ 815] 0 815 13018 110 20 0 0 rsyslogd
> [56317.529105] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> [56317.529112] [ 909] 0 909 4212 113 12 0 -1000 udevd
> [56317.529119] [ 910] 0 910 4212 113 12 0 -1000 udevd
> [56317.529126] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
> [56317.529133] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> [56317.529139] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> [56317.529146] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> [56317.529152] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> [56317.529159] [ 1016] 0 1016 86937 2043 155 0 0 apache2
> [56317.529166] [ 1026] 0 1026 4692 40 14 0 0 atd
> [56317.529177] [ 1054] 0 1054 5747 69 16 0 0 cron
> [56317.529184] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> [56317.529190] [ 1104] 106 1104 9630 153 23 0 0 ntpd
> [56317.529197] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> [56317.529203] [ 1130] 0 1130 18989 395 35 0 0 searchd
> [56317.529211] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> [56317.529220] [ 1152] 105 1152 10265 1088 24 0 0 unbound
> [56317.529228] [ 1172] 0 1172 1496 25 9 0 0 getty
> [56317.529237] [ 1173] 0 1173 1496 24 8 0 0 getty
> [56317.529245] [31557] 33 31557 88162 5277 161 0 0 apache2
> [56317.529252] [ 1031] 33 1031 87879 4726 161 0 0 apache2
> [56317.529261] [ 1579] 33 1579 87493 5027 158 0 0 apache2
> [56317.529267] [ 2901] 33 2901 87622 4707 158 0 0 apache2
> [56317.529274] [ 3447] 33 3447 35706 2055 71 0 0 apache2
> [56317.529281] [ 4559] 33 4559 87714 4945 160 0 0 apache2
> [56317.529287] [ 4560] 33 4560 35648 1981 71 0 0 apache2
> [56317.529296] [ 4561] 33 4561 87859 3840 159 0 0 apache2
> [56317.529304] [ 6205] 33 6205 87422 3053 154 0 0 apache2
> [56317.529311] [ 6776] 0 6776 10460 89 25 0 0 cron
> [56317.529317] [ 6780] 0 6780 1002 23 8 0 0 sh
> [56317.529324] [ 6784] 0 6784 7322 109 20 0 0 wget
> [56317.529331] [ 6847] 0 6847 10460 89 25 0 0 cron
> [56317.529338] [ 6856] 0 6856 1002 23 8 0 0 sh
> [56317.529345] [ 6862] 0 6862 7322 108 20 0 0 wget
> [56317.529351] [ 6924] 0 6924 10460 89 25 0 0 cron
> [56317.529361] [ 6928] 0 6928 1002 24 8 0 0 sh
> [56317.529367] [ 6932] 0 6932 7322 107 20 0 0 wget
> [56317.529374] [ 6948] 33 6948 35279 1532 70 0 0 apache2
> [56317.529381] [ 6991] 33 6991 86937 1966 137 0 0 apache2
> [56317.529388] [ 6992] 33 6992 86937 1966 137 0 0 apache2
> [56317.529394] [ 6993] 0 6993 10460 88 25 0 0 cron
> [56317.529403] [ 6994] 0 6994 10460 88 25 0 0 cron
> [56317.529410] [ 6995] 0 6995 10460 88 25 0 0 cron
> [56317.529416] [ 6999] 1337 6999 1002 23 8 0 0 sh
> [56317.529423] [ 7000] 0 7000 1002 24 8 0 0 sh
> [56317.529429] [ 7001] 0 7001 1002 23 8 0 0 sh
> [56317.529436] [ 7002] 1337 7002 1481 21 9 0 0 sleep
> [56317.529445] [ 7003] 0 7003 7322 110 20 0 0 wget
> [56317.529452] [ 7004] 0 7004 7322 110 20 0 0 wget
> [56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or sacrifice child
> [56317.529464] Killed process 31557 (apache2) total-vm:352648kB, anon-rss:11004kB, file-rss:10104kB
> [56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> [56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2
> [56680.628607] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006e3ffbb0
> [56680.628636] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001a8c8e0
> [56680.628645] ffffffff814d3d48 0000000000000000 00000000000201da 000000000000000b
> [56680.628654] Call Trace:
> [56680.628666] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56680.628675] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56680.628682] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56680.628692] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56680.628699] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56680.628706] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56680.628713] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [56680.628720] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56680.628727] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56680.628735] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56680.628741] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56680.628748] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56680.628756] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56680.628763] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56680.628768] Mem-Info:
> [56680.628771] Node 0 DMA per-cpu:
> [56680.628776] CPU 0: hi: 0, btch: 1 usd: 0
> [56680.628782] Node 0 DMA32 per-cpu:
> [56680.628788] CPU 0: hi: 186, btch: 31 usd: 115
> [56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0
> [56680.628796] active_file:156 inactive_file:1004 isolated_file:0
> [56680.628796] unevictable:0 dirty:0 writeback:0 unstable:0
> [56680.628796] free:3382 slab_reclaimable:103079 slab_unreclaimable:233653
> [56680.628796] mapped:3205 shmem:8327 pagetables:2067 bounce:0
> [56680.628796] free_cma:0
> [56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> [56680.628848] lowmem_reserve[]: 0 1973 1973 1973
> [56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB active_anon:78780kB inactive_anon:4132kB active_file:624kB inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12820kB shmem:33308kB slab_reclaimable:410684kB slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 all_unreclaimable? yes
> [56680.628883] lowmem_reserve[]: 0 0 0 0
> [56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> [56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB
> [56680.628945] 9494 total pagecache pages
> [56680.628949] 0 pages in swap cache
> [56680.628957] Swap cache stats: add 0, delete 0, find 0/0
> [56680.628962] Free swap = 0kB
> [56680.628965] Total swap = 0kB
> [56680.634908] 524287 pages RAM
> [56680.634916] 12090 pages reserved
> [56680.634919] 291027 pages shared
> [56680.634921] 431125 pages non-shared
> [56680.634924] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> [56680.634937] [ 330] 0 330 4213 114 12 0 -1000 udevd
> [56680.634943] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> [56680.634948] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> [56680.634953] [ 909] 0 909 4212 113 12 0 -1000 udevd
> [56680.634957] [ 910] 0 910 4212 113 12 0 -1000 udevd
> [56680.634961] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
> [56680.634966] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> [56680.634971] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> [56680.634975] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> [56680.634980] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> [56680.634985] [ 1016] 0 1016 86937 2014 155 0 0 apache2
> [56680.634990] [ 1026] 0 1026 4692 40 14 0 0 atd
> [56680.634995] [ 1054] 0 1054 5747 69 16 0 0 cron
> [56680.634999] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> [56680.635004] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> [56680.635008] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> [56680.635013] [ 1130] 0 1130 18989 400 35 0 0 searchd
> [56680.635018] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> [56680.635022] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> [56680.635027] [ 1172] 0 1172 1496 25 9 0 0 getty
> [56680.635144] [ 1173] 0 1173 1496 24 8 0 0 getty
> [56680.635150] [ 1031] 33 1031 87879 4814 161 0 0 apache2
> [56680.635155] [ 1579] 33 1579 87941 5548 159 0 0 apache2
> [56680.635160] [ 2901] 33 2901 87622 4692 158 0 0 apache2
> [56680.635164] [ 4559] 33 4559 87714 4947 160 0 0 apache2
> [56680.635169] [ 4561] 33 4561 87859 3852 159 0 0 apache2
> [56680.635174] [ 6205] 33 6205 87294 3684 155 0 0 apache2
> [56680.635179] [ 6991] 33 6991 87434 3395 152 0 0 apache2
> [56680.635183] [ 6992] 33 6992 87593 3185 148 0 0 apache2
> [56680.635188] [ 7289] 0 7289 10460 89 25 0 0 cron
> [56680.635192] [ 7297] 0 7297 1002 23 7 0 0 sh
> [56680.635197] [ 7303] 0 7303 7322 110 20 0 0 wget
> [56680.635202] [ 7363] 0 7363 10460 89 25 0 0 cron
> [56680.635206] [ 7374] 0 7374 1002 25 7 0 0 sh
> [56680.635213] [ 7378] 0 7378 7322 108 19 0 0 wget
> [56680.635218] [ 7437] 0 7437 10460 89 25 0 0 cron
> [56680.635222] [ 7438] 0 7438 10460 89 25 0 0 cron
> [56680.635227] [ 7439] 0 7439 10460 89 25 0 0 cron
> [56680.635231] [ 7443] 0 7443 1002 23 8 0 0 sh
> [56680.635235] [ 7444] 0 7444 1002 23 8 0 0 sh
> [56680.635239] [ 7445] 1337 7445 1002 24 7 0 0 sh
> [56680.635244] [ 7446] 1337 7446 1481 22 8 0 0 sleep
> [56680.635248] [ 7447] 0 7447 7322 109 19 0 0 wget
> [56680.635252] [ 7448] 0 7448 7322 107 19 0 0 wget
> [56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or sacrifice child
> [56680.635262] Killed process 1579 (apache2) total-vm:351764kB, anon-rss:12092kB, file-rss:10100kB
> [56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> [56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2
> [56734.778963] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006d133bb0
> [56734.778973] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800634686a0
> [56734.778982] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
> [56734.779011] Call Trace:
> [56734.779020] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56734.779026] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56734.779031] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56734.779037] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56734.779042] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56734.779046] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56734.779051] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [56734.779055] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56734.779060] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56734.779065] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56734.779069] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56734.779073] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56734.779079] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56734.779085] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> [56734.779089] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> [56734.779093] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> [56734.779097] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56734.779100] Mem-Info:
> [56734.779102] Node 0 DMA per-cpu:
> [56734.779106] CPU 0: hi: 0, btch: 1 usd: 0
> [56734.779108] Node 0 DMA32 per-cpu:
> [56734.779112] CPU 0: hi: 186, btch: 31 usd: 62
> [56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0
> [56734.779117] active_file:190 inactive_file:1109 isolated_file:0
> [56734.779117] unevictable:0 dirty:2 writeback:0 unstable:0
> [56734.779117] free:3395 slab_reclaimable:103429 slab_unreclaimable:233857
> [56734.779117] mapped:3186 shmem:8327 pagetables:2038 bounce:0
> [56734.779117] free_cma:0
> [56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> [56734.779256] lowmem_reserve[]: 0 1973 1973 1973
> [56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:75512kB inactive_anon:4132kB active_file:760kB inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB mapped:12744kB shmem:33308kB slab_reclaimable:412052kB slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 all_unreclaimable? yes
> [56734.779293] lowmem_reserve[]: 0 0 0 0
> [56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> [56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> [56734.779351] 9645 total pagecache pages
> [56734.779359] 0 pages in swap cache
> [56734.779363] Swap cache stats: add 0, delete 0, find 0/0
> [56734.779367] Free swap = 0kB
> [56734.779370] Total swap = 0kB
> [56734.785510] 524287 pages RAM
> [56734.785519] 12090 pages reserved
> [56734.785523] 286620 pages shared
> [56734.785526] 431108 pages non-shared
> [56734.785529] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> [56734.785544] [ 330] 0 330 4213 114 12 0 -1000 udevd
> [56734.785553] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> [56734.785560] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> [56734.785567] [ 909] 0 909 4212 113 12 0 -1000 udevd
> [56734.785573] [ 910] 0 910 4212 113 12 0 -1000 udevd
> [56734.785579] [ 911] 104 911 12487 208 27 0 0 zabbix_agentd
> [56734.785586] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> [56734.785593] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> [56734.785600] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> [56734.785607] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> [56734.785614] [ 1016] 0 1016 86937 2024 155 0 0 apache2
> [56734.785620] [ 1026] 0 1026 4692 40 14 0 0 atd
> [56734.785627] [ 1054] 0 1054 5747 70 16 0 0 cron
> [56734.785634] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> [56734.785641] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> [56734.785648] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> [56734.785654] [ 1130] 0 1130 18989 395 35 0 0 searchd
> [56734.785661] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> [56734.785665] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> [56734.785670] [ 1172] 0 1172 1496 25 9 0 0 getty
> [56734.785712] [ 1173] 0 1173 1496 24 8 0 0 getty
> [56734.785718] [ 1031] 33 1031 87879 4814 161 0 0 apache2
> [56734.785722] [ 2901] 33 2901 87622 4692 158 0 0 apache2
> [56734.785727] [ 4559] 33 4559 87714 4954 160 0 0 apache2
> [56734.785731] [ 4561] 33 4561 87859 3815 159 0 0 apache2
> [56734.785737] [ 6205] 33 6205 87294 3674 155 0 0 apache2
> [56734.785741] [ 6991] 33 6991 87434 3398 152 0 0 apache2
> [56734.785745] [ 6992] 33 6992 87593 3228 150 0 0 apache2
> [56734.785750] [ 7363] 0 7363 10460 89 25 0 0 cron
> [56734.785754] [ 7374] 0 7374 1002 25 7 0 0 sh
> [56734.785760] [ 7378] 0 7378 7322 108 19 0 0 wget
> [56734.785765] [ 7505] 0 7505 10460 89 25 0 0 cron
> [56734.785769] [ 7506] 0 7506 10460 89 25 0 0 cron
> [56734.785773] [ 7507] 0 7507 10460 89 25 0 0 cron
> [56734.785777] [ 7508] 0 7508 10460 89 25 0 0 cron
> [56734.785782] [ 7509] 0 7509 10460 89 25 0 0 cron
> [56734.785786] [ 7510] 0 7510 10460 89 25 0 0 cron
> [56734.785790] [ 7517] 0 7517 1002 24 8 0 0 sh
> [56734.785794] [ 7518] 0 7518 1002 24 8 0 0 sh
> [56734.785800] [ 7519] 0 7519 1002 24 8 0 0 sh
> [56734.785804] [ 7520] 0 7520 1002 23 8 0 0 sh
> [56734.785808] [ 7521] 1337 7521 1002 24 8 0 0 sh
> [56734.785813] [ 7522] 0 7522 1002 23 7 0 0 sh
> [56734.785817] [ 7523] 1337 7523 1481 21 8 0 0 sleep
> [56734.785822] [ 7524] 0 7524 7322 148 20 0 0 wget
> [56734.785826] [ 7525] 0 7525 7322 111 20 0 0 wget
> [56734.785830] [ 7526] 0 7526 7322 110 19 0 0 wget
> [56734.785834] [ 7527] 0 7527 7322 110 20 0 0 wget
> [56734.785839] [ 7528] 0 7528 1002 24 7 0 0 maxlifetime
> [56734.785846] [ 7530] 0 7530 16356 291 37 0 0 php5
> [56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or sacrifice child
> [56734.785854] Killed process 4559 (apache2) total-vm:350856kB, anon-rss:11596kB, file-rss:8220kB
> [56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> [56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2
> [56793.525928] 0000000000000000 ffffffff8134ab12 ffffffff8134f478 00000000000007c5
> [56793.525938] 0000000000000010 ffffffff81089ad6 ffffffff8134f09a ffff880001a127c0
> [56793.525946] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
> [56793.525976] Call Trace:
> [56793.525986] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56793.526012] [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6
> [56793.526042] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56793.526048] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56793.526057] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56793.526064] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56793.526070] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56793.526082] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [56793.526090] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56793.526097] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56793.526104] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56793.526111] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56793.526117] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56793.526125] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56793.526133] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> [56793.526142] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> [56793.526149] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> [56793.526155] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56793.526160] Mem-Info:
> [56793.526163] Node 0 DMA per-cpu:
> [56793.526171] CPU 0: hi: 0, btch: 1 usd: 0
> [56793.526175] Node 0 DMA32 per-cpu:
> [56793.526180] CPU 0: hi: 186, btch: 31 usd: 150
> [56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5
> [56793.526188] active_file:78 inactive_file:778 isolated_file:0
> [56793.526188] unevictable:0 dirty:0 writeback:0 unstable:0
> [56793.526188] free:3395 slab_reclaimable:104385 slab_unreclaimable:234096
> [56793.526188] mapped:2974 shmem:8327 pagetables:2047 bounce:0
> [56793.526188] free_cma:0
> [56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes
> [56793.526238] lowmem_reserve[]: 0 1973 1973 1973
> [56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:71364kB inactive_anon:4132kB active_file:312kB inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:11896kB shmem:33308kB slab_reclaimable:415876kB slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 all_unreclaimable? yes
> [56793.526277] lowmem_reserve[]: 0 0 0 0
> [56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> [56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> [56793.526343] 9197 total pagecache pages
> [56793.526347] 0 pages in swap cache
> [56793.526352] Swap cache stats: add 0, delete 0, find 0/0
> [56793.526356] Free swap = 0kB
> [56793.526359] Total swap = 0kB
> [56793.532348] 524287 pages RAM
> [56793.532355] 12090 pages reserved
> [56793.532358] 284892 pages shared
> [56793.532360] 430988 pages non-shared
> [56793.532363] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> [56793.532376] [ 330] 0 330 4213 114 12 0 -1000 udevd
> [56793.532381] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> [56793.532386] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> [56793.532391] [ 909] 0 909 4212 113 12 0 -1000 udevd
> [56793.532396] [ 910] 0 910 4212 113 12 0 -1000 udevd
> [56793.532400] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
> [56793.532405] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> [56793.532409] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> [56793.532414] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> [56793.532418] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> [56793.532424] [ 1016] 0 1016 86937 2028 155 0 0 apache2
> [56793.532428] [ 1026] 0 1026 4692 40 14 0 0 atd
> [56793.532433] [ 1054] 0 1054 5747 70 16 0 0 cron
> [56793.532437] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> [56793.532442] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> [56793.532447] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> [56793.532451] [ 1130] 0 1130 18989 374 35 0 0 searchd
> [56793.532456] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> [56793.532461] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> [56793.532467] [ 1172] 0 1172 1496 25 9 0 0 getty
> [56793.532484] [ 1173] 0 1173 1496 24 8 0 0 getty
> [56793.532491] [ 1031] 33 1031 87879 4791 161 0 0 apache2
> [56793.532497] [ 2901] 33 2901 87622 4723 158 0 0 apache2
> [56793.532504] [ 4561] 33 4561 87859 3866 159 0 0 apache2
> [56793.532510] [ 6205] 33 6205 87294 3663 155 0 0 apache2
> [56793.532515] [ 6991] 33 6991 87434 3398 152 0 0 apache2
> [56793.532522] [ 6992] 33 6992 87593 3228 150 0 0 apache2
> [56793.532526] [ 7363] 0 7363 10460 89 25 0 0 cron
> [56793.532531] [ 7374] 0 7374 1002 25 7 0 0 sh
> [56793.532537] [ 7378] 0 7378 7322 108 19 0 0 wget
> [56793.532547] [ 7555] 33 7555 86937 1967 137 0 0 apache2
> [56793.532553] [ 7593] 0 7593 10460 88 25 0 0 cron
> [56793.532559] [ 7594] 0 7594 10460 88 25 0 0 cron
> [56793.532564] [ 7595] 0 7595 10460 88 25 0 0 cron
> [56793.532568] [ 7596] 0 7596 10460 88 25 0 0 cron
> [56793.532573] [ 7597] 0 7597 10460 88 25 0 0 cron
> [56793.532580] [ 7598] 0 7598 10460 88 25 0 0 cron
> [56793.532584] [ 7599] 0 7599 10460 88 25 0 0 cron
> [56793.532588] [ 7607] 0 7607 1002 25 8 0 0 sh
> [56793.532593] [ 7608] 1337 7608 1002 24 8 0 0 sh
> [56793.532597] [ 7609] 0 7609 1002 26 7 0 0 sh
> [56793.532601] [ 7610] 0 7610 1002 24 7 0 0 sh
> [56793.532606] [ 7611] 0 7611 1002 25 7 0 0 sh
> [56793.532611] [ 7612] 0 7612 1002 23 8 0 0 sh
> [56793.532615] [ 7613] 0 7613 1002 25 7 0 0 sh
> [56793.532620] [ 7614] 0 7614 7322 156 20 0 0 wget
> [56793.532624] [ 7615] 1337 7615 1481 19 8 0 0 sleep
> [56793.532628] [ 7616] 0 7616 7322 110 19 0 0 wget
> [56793.532632] [ 7617] 0 7617 7322 110 20 0 0 wget
> [56793.532637] [ 7618] 0 7618 7322 109 19 0 0 wget
> [56793.532641] [ 7619] 0 7619 7322 109 20 0 0 wget
> [56793.532646] [ 7620] 0 7620 7322 109 20 0 0 wget
> [56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or sacrifice child
> [56793.532656] Killed process 1031 (apache2) total-vm:351516kB, anon-rss:9624kB, file-rss:9540kB
>
> ...
>
> [57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> [57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted 2.6.50-dae-xen #2
> [57131.763847] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff880073e21bb0
> [57131.763858] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800788aea00
> [57131.763867] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000004
> [57131.763885] Call Trace:
> [57131.763896] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [57131.763906] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [57131.763913] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [57131.763921] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [57131.763947] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [57131.763955] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [57131.763962] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [57131.763969] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [57131.763977] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [57131.763985] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [57131.763991] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [57131.763997] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [57131.764006] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [57131.764017] [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182
> [57131.764023] [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7
> [57131.764030] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [57131.764034] Mem-Info:
> [57131.764037] Node 0 DMA per-cpu:
> [57131.764041] CPU 0: hi: 0, btch: 1 usd: 0
> [57131.764045] Node 0 DMA32 per-cpu:
> [57131.764049] CPU 0: hi: 186, btch: 31 usd: 62
> [57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0
> [57131.764076] active_file:288 inactive_file:1121 isolated_file:0
> [57131.764076] unevictable:0 dirty:0 writeback:0 unstable:0
> [57131.764076] free:3369 slab_reclaimable:109088 slab_unreclaimable:235420
> [57131.764076] mapped:358 shmem:8328 pagetables:821 bounce:0
> [57131.764076] free_cma:0
> [57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> [57131.764115] lowmem_reserve[]: 0 1973 1973 1973
> [57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB active_anon:46972kB inactive_anon:4104kB active_file:1152kB inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:1432kB shmem:33312kB slab_reclaimable:434592kB slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 all_unreclaimable? yes
> [57131.764151] lowmem_reserve[]: 0 0 0 0
> [57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> [57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5548kB
> [57131.764242] 9744 total pagecache pages
> [57131.764253] 0 pages in swap cache
> [57131.764256] Swap cache stats: add 0, delete 0, find 0/0
> [57131.764263] Free swap = 0kB
> [57131.764267] Total swap = 0kB
> [57131.770252] 524287 pages RAM
> [57131.770260] 12090 pages reserved
> [57131.770262] 265440 pages shared
> [57131.770265] 433233 pages non-shared
> [57131.770267] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> [57131.770280] [ 330] 0 330 4213 114 12 0 -1000 udevd
> [57131.770285] [ 815] 0 815 13048 150 20 0 0 rsyslogd
> [57131.770290] [ 853] 104 853 12486 99 27 0 0 zabbix_agentd
> [57131.770295] [ 909] 0 909 4212 113 12 0 -1000 udevd
> [57131.770299] [ 910] 0 910 4212 113 12 0 -1000 udevd
> [57131.770305] [ 1016] 0 1016 86937 2031 155 0 0 apache2
> [57131.770310] [ 1054] 0 1054 5747 69 16 0 0 cron
> [57131.770314] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> [57131.770319] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> [57131.770323] [ 1130] 0 1130 18989 374 35 0 0 searchd
> [57131.770328] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> [57131.770334] [ 7363] 0 7363 10460 89 25 0 0 cron
> [57131.770339] [ 7374] 0 7374 1002 25 7 0 0 sh
> [57131.770343] [ 7378] 0 7378 7322 108 19 0 0 wget
> [57131.770348] [ 8003] 0 8003 10460 88 25 0 0 cron
> [57131.770352] [ 8006] 0 8006 10460 88 25 0 0 cron
> [57131.770357] [ 8009] 0 8009 10460 88 25 0 0 cron
> [57131.770361] [ 8020] 1337 8020 1002 23 7 0 0 sh
> [57131.770366] [ 8023] 0 8023 1002 23 6 0 0 sh
> [57131.770370] [ 8024] 0 8024 1002 24 7 0 0 sh
> [57131.770399] [ 8030] 0 8030 7322 108 19 0 0 wget
> [57131.770406] [ 8035] 0 8035 7322 109 20 0 0 wget
> [57131.770410] [ 8051] 0 8051 1002 41 8 0 0 rc
> [57131.770414] [ 8054] 0 8054 3085 60 11 0 0 startpar
> [57131.770419] [ 8055] 1337 8055 1002 23 8 0 0 zabbix-launch-t
> [57131.770424] [ 8057] 0 8057 1002 40 7 0 0 apache2
> [57131.770430] [ 8067] 1337 8067 2841 76 9 0 0 apache
> [57131.770436] [ 8084] 0 8084 1002 24 8 0 0 apache2ctl
> [57131.770443] [ 8092] 0 8092 26561 288 48 0 0 apache2
> [57131.770450] [ 8115] 0 8115 1002 22 8 0 0 zabbix-agent
> [57131.770457] [ 8116] 0 8116 980 32 8 0 0 start-stop-daem
> [57131.770461] [ 8158] 1337 8158 10130 168 22 0 0 zabbix_sender
> [57131.770466] [ 8161] 33 8161 86937 1967 137 0 0 apache2
> [57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or sacrifice child
> [57131.770475] Killed process 8161 (apache2) total-vm:347748kB, anon-rss:7868kB, file-rss:0kB
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 8:29 ` [Xen-users] " Ian Campbell
@ 2013-08-07 8:58 ` Olivier Bonvalet
2013-08-07 9:35 ` Olivier Bonvalet
0 siblings, 1 reply; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 8:58 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, xen-devel
Le mercredi 07 août 2013 à 09:29 +0100, Ian Campbell a écrit :
> On Wed, 2013-08-07 at 02:02 +0200, Olivier Bonvalet wrote:
> > Hi,
> >
> > I have a memory problem on several PV domU, I *suppose* since the
> > upgrade to 3.10.4 kernel (and still occur with 3.10.5).
> > I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy
> > with a vanilla 3.9.11 kernel.
> >
> > The problem : once a domU throw oom-killer, then it loop until all
> > processes will be killed, then the kernel abort and shutdown the VM.
> >
> > I'm not sure how to read the OOM-killer messages about memory state, so
> > I copy/paste a list of them here, about a 2GB RAM domU :
>
> Does /proc/meminfo on the VM show that it actually has 2GB?
>
Now yes. I will follow it, for the next time.
> > [56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
> > [56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2
>
> This appears to show a 2.6.50 kernel, whatever that is, and not 3.10.x?
>
Yes it's a 3.10.5 here, it was renamed for compatibility with some old
code.
> > [56313.921413] 0000000000000000 ffffffff8134ab12 ffff880079cd2120 0000000000000000
> > [56313.921424] 00000000000280da ffffffff81089ad6 ffffffff8134f09a ffff88007773bba0
> > [56313.921433] ffffffff814d3d48 0000000000000000 00000000000280da 000000000000000b
> > [56313.921443] Call Trace:
> > [56313.921454] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56313.921462] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56313.921470] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56313.921480] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56313.921486] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56313.921493] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56313.921502] [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc
> > [56313.921510] [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21
> > [56313.921516] [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed
> > [56313.921538] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56313.921547] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56313.921560] [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14
> > [56313.921566] [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133
> > [56313.921572] [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14
> > [56313.921579] [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3
> > [56313.921587] [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b
> > [56313.921593] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56313.921600] [<ffffffff81101c75>] ? seq_read+0xb8/0x31a
> > [56313.921606] [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40
> > [56313.921612] [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a
> > [56313.921632] [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74
> > [56313.921644] [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0
> > [56313.921650] [<ffffffff810e9e38>] ? SyS_read+0x41/0x70
> > [56313.921656] [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b
> > [56313.921662] Mem-Info:
> > [56313.921671] Node 0 DMA per-cpu:
> > [56313.921676] CPU 0: hi: 0, btch: 1 usd: 0
> > [56313.921680] Node 0 DMA32 per-cpu:
> > [56313.921684] CPU 0: hi: 186, btch: 31 usd: 83
> > [56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1
> > [56313.921693] active_file:153 inactive_file:849 isolated_file:0
> > [56313.921693] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56313.921693] free:3392 slab_reclaimable:101577 slab_unreclaimable:232545
> > [56313.921693] mapped:3613 shmem:8327 pagetables:2848 bounce:0
> > [56313.921693] free_cma:0
> > [56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56313.921726] lowmem_reserve[]: 0 1973 1973 1973
> > [56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB active_anon:90052kB inactive_anon:4132kB active_file:612kB inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:14452kB shmem:33308kB slab_reclaimable:404740kB slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 all_unreclaimable? yes
> > [56313.921749] lowmem_reserve[]: 0 0 0 0
> > [56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
> > [56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB
> > [56313.921791] 9341 total pagecache pages
> > [56313.921794] 0 pages in swap cache
> > [56313.921797] Swap cache stats: add 0, delete 0, find 0/0
> > [56313.921799] Free swap = 0kB
> > [56313.921802] Total swap = 0kB
> > [56313.927968] 524287 pages RAM
> > [56313.927976] 12090 pages reserved
> > [56313.927978] 307132 pages shared
> > [56313.927980] 430905 pages non-shared
> > [56313.927983] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56313.927996] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56313.928001] [ 815] 0 815 13018 114 20 0 0 rsyslogd
> > [56313.928006] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56313.928011] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56313.928015] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56313.928020] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
> > [56313.928025] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56313.928029] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56313.928034] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56313.928038] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56313.928043] [ 1016] 0 1016 86937 2033 155 0 0 apache2
> > [56313.928048] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56313.928057] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [56313.928062] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56313.928067] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56313.928072] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56313.928077] [ 1130] 0 1130 18989 374 35 0 0 searchd
> > [56313.928081] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56313.928086] [ 1152] 105 1152 10265 1088 24 0 0 unbound
> > [56313.928090] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56313.928150] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56313.928157] [31557] 33 31557 88162 5277 161 0 0 apache2
> > [56313.928163] [31998] 33 31998 88044 5689 162 0 0 apache2
> > [56313.928168] [ 1031] 33 1031 87879 4726 161 0 0 apache2
> > [56313.928172] [ 1579] 33 1579 87493 5027 158 0 0 apache2
> > [56313.928177] [ 2901] 33 2901 87622 4707 158 0 0 apache2
> > [56313.928182] [ 3447] 33 3447 82447 4550 150 0 0 apache2
> > [56313.928186] [ 4559] 33 4559 87714 4945 160 0 0 apache2
> > [56313.928191] [ 4560] 33 4560 82914 4906 150 0 0 apache2
> > [56313.928195] [ 4561] 33 4561 87859 3840 159 0 0 apache2
> > [56313.928200] [ 6205] 33 6205 87422 3053 154 0 0 apache2
> > [56313.928204] [ 6776] 0 6776 10460 89 25 0 0 cron
> > [56313.928229] [ 6780] 0 6780 1002 23 8 0 0 sh
> > [56313.928236] [ 6784] 0 6784 7322 109 20 0 0 wget
> > [56313.928241] [ 6847] 0 6847 10460 89 25 0 0 cron
> > [56313.928246] [ 6856] 0 6856 1002 23 8 0 0 sh
> > [56313.928253] [ 6862] 0 6862 7322 108 20 0 0 wget
> > [56313.928258] [ 6924] 0 6924 10460 89 25 0 0 cron
> > [56313.928263] [ 6928] 0 6928 1002 24 8 0 0 sh
> > [56313.928267] [ 6932] 0 6932 7322 107 20 0 0 wget
> > [56313.928272] [ 6948] 33 6948 82020 2026 135 0 0 apache2
> > [56313.928277] [ 6991] 33 6991 86937 1966 137 0 0 apache2
> > [56313.928281] [ 6992] 33 6992 86937 1966 137 0 0 apache2
> > [56313.928285] [ 6993] 0 6993 10460 106 25 0 0 cron
> > [56313.928290] [ 6994] 0 6994 10460 106 25 0 0 cron
> > [56313.928294] [ 6995] 0 6995 10460 106 25 0 0 cron
> > [56313.928298] [ 6999] 1337 6999 10459 183 23 0 0 cron
> > [56313.928302] [ 7000] 0 7000 10459 182 23 0 0 cron
> > [56313.928306] [ 7001] 0 7001 10460 182 23 0 0 cron
> > [56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or sacrifice child
> > [56313.928315] Killed process 31998 (apache2) total-vm:352176kB, anon-rss:10848kB, file-rss:11908kB
> > [56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, oom_score_adj=0
> > [56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2
> > [56317.522115] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff8800793078e8
> > [56317.522125] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001802120
> > [56317.522133] ffffffff814d3d48 0000000000000000 00000000002840d0 000000000000000a
> > [56317.522142] Call Trace:
> > [56317.522153] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56317.522161] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56317.522168] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56317.522177] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56317.522184] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56317.522190] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56317.522197] [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5
> > [56317.522203] [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b
> > [56317.522211] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> > [56317.522219] [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d
> > [56317.522226] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> > [56317.522232] [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f
> > [56317.522238] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> > [56317.522245] [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92
> > [56317.522251] [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31
> > [56317.522258] [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105
> > [56317.522264] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> > [56317.522271] [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab
> > [56317.522277] [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b
> > [56317.522283] [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f
> > [56317.522289] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56317.522296] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56317.522304] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56317.522310] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56317.522440] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56317.522449] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56317.522457] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> > [56317.522464] [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20
> > [56317.522471] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> > [56317.522479] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
> > [56317.522498] [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72
> > [56317.522503] [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3
> > [56317.522508] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> > [56317.522512] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31
> > [56317.522517] [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55
> > [56317.522522] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56317.522526] Mem-Info:
> > [56317.522529] Node 0 DMA per-cpu:
> > [56317.522540] CPU 0: hi: 0, btch: 1 usd: 0
> > [56317.522544] Node 0 DMA32 per-cpu:
> > [56317.522549] CPU 0: hi: 186, btch: 31 usd: 90
> > [56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0
> > [56317.522556] active_file:95 inactive_file:793 isolated_file:0
> > [56317.522556] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56317.522556] free:3378 slab_reclaimable:101712 slab_unreclaimable:232523
> > [56317.522556] mapped:3140 shmem:8327 pagetables:2478 bounce:0
> > [56317.522556] free_cma:0
> > [56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56317.522590] lowmem_reserve[]: 0 1973 1973 1973
> > [56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB active_anon:91128kB inactive_anon:4132kB active_file:380kB inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12560kB shmem:33308kB slab_reclaimable:405280kB slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 all_unreclaimable? yes
> > [56317.522621] lowmem_reserve[]: 0 0 0 0
> > [56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB
> > [56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB
> > [56317.522660] 9223 total pagecache pages
> > [56317.522663] 0 pages in swap cache
> > [56317.522668] Swap cache stats: add 0, delete 0, find 0/0
> > [56317.522671] Free swap = 0kB
> > [56317.522673] Total swap = 0kB
> > [56317.529054] 524287 pages RAM
> > [56317.529064] 12090 pages reserved
> > [56317.529068] 294744 pages shared
> > [56317.529071] 431526 pages non-shared
> > [56317.529075] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56317.529090] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56317.529098] [ 815] 0 815 13018 110 20 0 0 rsyslogd
> > [56317.529105] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56317.529112] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56317.529119] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56317.529126] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
> > [56317.529133] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56317.529139] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56317.529146] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56317.529152] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56317.529159] [ 1016] 0 1016 86937 2043 155 0 0 apache2
> > [56317.529166] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56317.529177] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [56317.529184] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56317.529190] [ 1104] 106 1104 9630 153 23 0 0 ntpd
> > [56317.529197] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56317.529203] [ 1130] 0 1130 18989 395 35 0 0 searchd
> > [56317.529211] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56317.529220] [ 1152] 105 1152 10265 1088 24 0 0 unbound
> > [56317.529228] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56317.529237] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56317.529245] [31557] 33 31557 88162 5277 161 0 0 apache2
> > [56317.529252] [ 1031] 33 1031 87879 4726 161 0 0 apache2
> > [56317.529261] [ 1579] 33 1579 87493 5027 158 0 0 apache2
> > [56317.529267] [ 2901] 33 2901 87622 4707 158 0 0 apache2
> > [56317.529274] [ 3447] 33 3447 35706 2055 71 0 0 apache2
> > [56317.529281] [ 4559] 33 4559 87714 4945 160 0 0 apache2
> > [56317.529287] [ 4560] 33 4560 35648 1981 71 0 0 apache2
> > [56317.529296] [ 4561] 33 4561 87859 3840 159 0 0 apache2
> > [56317.529304] [ 6205] 33 6205 87422 3053 154 0 0 apache2
> > [56317.529311] [ 6776] 0 6776 10460 89 25 0 0 cron
> > [56317.529317] [ 6780] 0 6780 1002 23 8 0 0 sh
> > [56317.529324] [ 6784] 0 6784 7322 109 20 0 0 wget
> > [56317.529331] [ 6847] 0 6847 10460 89 25 0 0 cron
> > [56317.529338] [ 6856] 0 6856 1002 23 8 0 0 sh
> > [56317.529345] [ 6862] 0 6862 7322 108 20 0 0 wget
> > [56317.529351] [ 6924] 0 6924 10460 89 25 0 0 cron
> > [56317.529361] [ 6928] 0 6928 1002 24 8 0 0 sh
> > [56317.529367] [ 6932] 0 6932 7322 107 20 0 0 wget
> > [56317.529374] [ 6948] 33 6948 35279 1532 70 0 0 apache2
> > [56317.529381] [ 6991] 33 6991 86937 1966 137 0 0 apache2
> > [56317.529388] [ 6992] 33 6992 86937 1966 137 0 0 apache2
> > [56317.529394] [ 6993] 0 6993 10460 88 25 0 0 cron
> > [56317.529403] [ 6994] 0 6994 10460 88 25 0 0 cron
> > [56317.529410] [ 6995] 0 6995 10460 88 25 0 0 cron
> > [56317.529416] [ 6999] 1337 6999 1002 23 8 0 0 sh
> > [56317.529423] [ 7000] 0 7000 1002 24 8 0 0 sh
> > [56317.529429] [ 7001] 0 7001 1002 23 8 0 0 sh
> > [56317.529436] [ 7002] 1337 7002 1481 21 9 0 0 sleep
> > [56317.529445] [ 7003] 0 7003 7322 110 20 0 0 wget
> > [56317.529452] [ 7004] 0 7004 7322 110 20 0 0 wget
> > [56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or sacrifice child
> > [56317.529464] Killed process 31557 (apache2) total-vm:352648kB, anon-rss:11004kB, file-rss:10104kB
> > [56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2
> > [56680.628607] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006e3ffbb0
> > [56680.628636] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001a8c8e0
> > [56680.628645] ffffffff814d3d48 0000000000000000 00000000000201da 000000000000000b
> > [56680.628654] Call Trace:
> > [56680.628666] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56680.628675] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56680.628682] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56680.628692] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56680.628699] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56680.628706] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56680.628713] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [56680.628720] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56680.628727] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56680.628735] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56680.628741] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56680.628748] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56680.628756] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56680.628763] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56680.628768] Mem-Info:
> > [56680.628771] Node 0 DMA per-cpu:
> > [56680.628776] CPU 0: hi: 0, btch: 1 usd: 0
> > [56680.628782] Node 0 DMA32 per-cpu:
> > [56680.628788] CPU 0: hi: 186, btch: 31 usd: 115
> > [56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0
> > [56680.628796] active_file:156 inactive_file:1004 isolated_file:0
> > [56680.628796] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56680.628796] free:3382 slab_reclaimable:103079 slab_unreclaimable:233653
> > [56680.628796] mapped:3205 shmem:8327 pagetables:2067 bounce:0
> > [56680.628796] free_cma:0
> > [56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56680.628848] lowmem_reserve[]: 0 1973 1973 1973
> > [56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB active_anon:78780kB inactive_anon:4132kB active_file:624kB inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12820kB shmem:33308kB slab_reclaimable:410684kB slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 all_unreclaimable? yes
> > [56680.628883] lowmem_reserve[]: 0 0 0 0
> > [56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB
> > [56680.628945] 9494 total pagecache pages
> > [56680.628949] 0 pages in swap cache
> > [56680.628957] Swap cache stats: add 0, delete 0, find 0/0
> > [56680.628962] Free swap = 0kB
> > [56680.628965] Total swap = 0kB
> > [56680.634908] 524287 pages RAM
> > [56680.634916] 12090 pages reserved
> > [56680.634919] 291027 pages shared
> > [56680.634921] 431125 pages non-shared
> > [56680.634924] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56680.634937] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56680.634943] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> > [56680.634948] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56680.634953] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56680.634957] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56680.634961] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd
> > [56680.634966] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56680.634971] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56680.634975] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56680.634980] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56680.634985] [ 1016] 0 1016 86937 2014 155 0 0 apache2
> > [56680.634990] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56680.634995] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [56680.634999] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56680.635004] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56680.635008] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56680.635013] [ 1130] 0 1130 18989 400 35 0 0 searchd
> > [56680.635018] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56680.635022] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> > [56680.635027] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56680.635144] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56680.635150] [ 1031] 33 1031 87879 4814 161 0 0 apache2
> > [56680.635155] [ 1579] 33 1579 87941 5548 159 0 0 apache2
> > [56680.635160] [ 2901] 33 2901 87622 4692 158 0 0 apache2
> > [56680.635164] [ 4559] 33 4559 87714 4947 160 0 0 apache2
> > [56680.635169] [ 4561] 33 4561 87859 3852 159 0 0 apache2
> > [56680.635174] [ 6205] 33 6205 87294 3684 155 0 0 apache2
> > [56680.635179] [ 6991] 33 6991 87434 3395 152 0 0 apache2
> > [56680.635183] [ 6992] 33 6992 87593 3185 148 0 0 apache2
> > [56680.635188] [ 7289] 0 7289 10460 89 25 0 0 cron
> > [56680.635192] [ 7297] 0 7297 1002 23 7 0 0 sh
> > [56680.635197] [ 7303] 0 7303 7322 110 20 0 0 wget
> > [56680.635202] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [56680.635206] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [56680.635213] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [56680.635218] [ 7437] 0 7437 10460 89 25 0 0 cron
> > [56680.635222] [ 7438] 0 7438 10460 89 25 0 0 cron
> > [56680.635227] [ 7439] 0 7439 10460 89 25 0 0 cron
> > [56680.635231] [ 7443] 0 7443 1002 23 8 0 0 sh
> > [56680.635235] [ 7444] 0 7444 1002 23 8 0 0 sh
> > [56680.635239] [ 7445] 1337 7445 1002 24 7 0 0 sh
> > [56680.635244] [ 7446] 1337 7446 1481 22 8 0 0 sleep
> > [56680.635248] [ 7447] 0 7447 7322 109 19 0 0 wget
> > [56680.635252] [ 7448] 0 7448 7322 107 19 0 0 wget
> > [56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or sacrifice child
> > [56680.635262] Killed process 1579 (apache2) total-vm:351764kB, anon-rss:12092kB, file-rss:10100kB
> > [56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2
> > [56734.778963] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006d133bb0
> > [56734.778973] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800634686a0
> > [56734.778982] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
> > [56734.779011] Call Trace:
> > [56734.779020] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56734.779026] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56734.779031] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56734.779037] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56734.779042] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56734.779046] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56734.779051] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [56734.779055] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56734.779060] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56734.779065] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56734.779069] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56734.779073] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56734.779079] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56734.779085] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> > [56734.779089] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> > [56734.779093] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> > [56734.779097] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56734.779100] Mem-Info:
> > [56734.779102] Node 0 DMA per-cpu:
> > [56734.779106] CPU 0: hi: 0, btch: 1 usd: 0
> > [56734.779108] Node 0 DMA32 per-cpu:
> > [56734.779112] CPU 0: hi: 186, btch: 31 usd: 62
> > [56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0
> > [56734.779117] active_file:190 inactive_file:1109 isolated_file:0
> > [56734.779117] unevictable:0 dirty:2 writeback:0 unstable:0
> > [56734.779117] free:3395 slab_reclaimable:103429 slab_unreclaimable:233857
> > [56734.779117] mapped:3186 shmem:8327 pagetables:2038 bounce:0
> > [56734.779117] free_cma:0
> > [56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [56734.779256] lowmem_reserve[]: 0 1973 1973 1973
> > [56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:75512kB inactive_anon:4132kB active_file:760kB inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB mapped:12744kB shmem:33308kB slab_reclaimable:412052kB slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 all_unreclaimable? yes
> > [56734.779293] lowmem_reserve[]: 0 0 0 0
> > [56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> > [56734.779351] 9645 total pagecache pages
> > [56734.779359] 0 pages in swap cache
> > [56734.779363] Swap cache stats: add 0, delete 0, find 0/0
> > [56734.779367] Free swap = 0kB
> > [56734.779370] Total swap = 0kB
> > [56734.785510] 524287 pages RAM
> > [56734.785519] 12090 pages reserved
> > [56734.785523] 286620 pages shared
> > [56734.785526] 431108 pages non-shared
> > [56734.785529] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56734.785544] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56734.785553] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> > [56734.785560] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56734.785567] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56734.785573] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56734.785579] [ 911] 104 911 12487 208 27 0 0 zabbix_agentd
> > [56734.785586] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56734.785593] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56734.785600] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56734.785607] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56734.785614] [ 1016] 0 1016 86937 2024 155 0 0 apache2
> > [56734.785620] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56734.785627] [ 1054] 0 1054 5747 70 16 0 0 cron
> > [56734.785634] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56734.785641] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56734.785648] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56734.785654] [ 1130] 0 1130 18989 395 35 0 0 searchd
> > [56734.785661] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56734.785665] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> > [56734.785670] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56734.785712] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56734.785718] [ 1031] 33 1031 87879 4814 161 0 0 apache2
> > [56734.785722] [ 2901] 33 2901 87622 4692 158 0 0 apache2
> > [56734.785727] [ 4559] 33 4559 87714 4954 160 0 0 apache2
> > [56734.785731] [ 4561] 33 4561 87859 3815 159 0 0 apache2
> > [56734.785737] [ 6205] 33 6205 87294 3674 155 0 0 apache2
> > [56734.785741] [ 6991] 33 6991 87434 3398 152 0 0 apache2
> > [56734.785745] [ 6992] 33 6992 87593 3228 150 0 0 apache2
> > [56734.785750] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [56734.785754] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [56734.785760] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [56734.785765] [ 7505] 0 7505 10460 89 25 0 0 cron
> > [56734.785769] [ 7506] 0 7506 10460 89 25 0 0 cron
> > [56734.785773] [ 7507] 0 7507 10460 89 25 0 0 cron
> > [56734.785777] [ 7508] 0 7508 10460 89 25 0 0 cron
> > [56734.785782] [ 7509] 0 7509 10460 89 25 0 0 cron
> > [56734.785786] [ 7510] 0 7510 10460 89 25 0 0 cron
> > [56734.785790] [ 7517] 0 7517 1002 24 8 0 0 sh
> > [56734.785794] [ 7518] 0 7518 1002 24 8 0 0 sh
> > [56734.785800] [ 7519] 0 7519 1002 24 8 0 0 sh
> > [56734.785804] [ 7520] 0 7520 1002 23 8 0 0 sh
> > [56734.785808] [ 7521] 1337 7521 1002 24 8 0 0 sh
> > [56734.785813] [ 7522] 0 7522 1002 23 7 0 0 sh
> > [56734.785817] [ 7523] 1337 7523 1481 21 8 0 0 sleep
> > [56734.785822] [ 7524] 0 7524 7322 148 20 0 0 wget
> > [56734.785826] [ 7525] 0 7525 7322 111 20 0 0 wget
> > [56734.785830] [ 7526] 0 7526 7322 110 19 0 0 wget
> > [56734.785834] [ 7527] 0 7527 7322 110 20 0 0 wget
> > [56734.785839] [ 7528] 0 7528 1002 24 7 0 0 maxlifetime
> > [56734.785846] [ 7530] 0 7530 16356 291 37 0 0 php5
> > [56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or sacrifice child
> > [56734.785854] Killed process 4559 (apache2) total-vm:350856kB, anon-rss:11596kB, file-rss:8220kB
> > [56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2
> > [56793.525928] 0000000000000000 ffffffff8134ab12 ffffffff8134f478 00000000000007c5
> > [56793.525938] 0000000000000010 ffffffff81089ad6 ffffffff8134f09a ffff880001a127c0
> > [56793.525946] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009
> > [56793.525976] Call Trace:
> > [56793.525986] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [56793.526012] [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6
> > [56793.526042] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [56793.526048] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [56793.526057] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [56793.526064] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [56793.526070] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [56793.526082] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [56793.526090] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [56793.526097] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [56793.526104] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [56793.526111] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [56793.526117] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [56793.526125] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [56793.526133] [<ffffffff81113891>] ? fsnotify+0x261/0x291
> > [56793.526142] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> > [56793.526149] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> > [56793.526155] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [56793.526160] Mem-Info:
> > [56793.526163] Node 0 DMA per-cpu:
> > [56793.526171] CPU 0: hi: 0, btch: 1 usd: 0
> > [56793.526175] Node 0 DMA32 per-cpu:
> > [56793.526180] CPU 0: hi: 186, btch: 31 usd: 150
> > [56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5
> > [56793.526188] active_file:78 inactive_file:778 isolated_file:0
> > [56793.526188] unevictable:0 dirty:0 writeback:0 unstable:0
> > [56793.526188] free:3395 slab_reclaimable:104385 slab_unreclaimable:234096
> > [56793.526188] mapped:2974 shmem:8327 pagetables:2047 bounce:0
> > [56793.526188] free_cma:0
> > [56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes
> > [56793.526238] lowmem_reserve[]: 0 1973 1973 1973
> > [56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:71364kB inactive_anon:4132kB active_file:312kB inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:11896kB shmem:33308kB slab_reclaimable:415876kB slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 all_unreclaimable? yes
> > [56793.526277] lowmem_reserve[]: 0 0 0 0
> > [56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> > [56793.526343] 9197 total pagecache pages
> > [56793.526347] 0 pages in swap cache
> > [56793.526352] Swap cache stats: add 0, delete 0, find 0/0
> > [56793.526356] Free swap = 0kB
> > [56793.526359] Total swap = 0kB
> > [56793.532348] 524287 pages RAM
> > [56793.532355] 12090 pages reserved
> > [56793.532358] 284892 pages shared
> > [56793.532360] 430988 pages non-shared
> > [56793.532363] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [56793.532376] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [56793.532381] [ 815] 0 815 13018 138 20 0 0 rsyslogd
> > [56793.532386] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd
> > [56793.532391] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [56793.532396] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [56793.532400] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd
> > [56793.532405] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd
> > [56793.532409] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd
> > [56793.532414] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd
> > [56793.532418] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd
> > [56793.532424] [ 1016] 0 1016 86937 2028 155 0 0 apache2
> > [56793.532428] [ 1026] 0 1026 4692 40 14 0 0 atd
> > [56793.532433] [ 1054] 0 1054 5747 70 16 0 0 cron
> > [56793.532437] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [56793.532442] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [56793.532447] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send
> > [56793.532451] [ 1130] 0 1130 18989 374 35 0 0 searchd
> > [56793.532456] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [56793.532461] [ 1152] 105 1152 10265 1099 24 0 0 unbound
> > [56793.532467] [ 1172] 0 1172 1496 25 9 0 0 getty
> > [56793.532484] [ 1173] 0 1173 1496 24 8 0 0 getty
> > [56793.532491] [ 1031] 33 1031 87879 4791 161 0 0 apache2
> > [56793.532497] [ 2901] 33 2901 87622 4723 158 0 0 apache2
> > [56793.532504] [ 4561] 33 4561 87859 3866 159 0 0 apache2
> > [56793.532510] [ 6205] 33 6205 87294 3663 155 0 0 apache2
> > [56793.532515] [ 6991] 33 6991 87434 3398 152 0 0 apache2
> > [56793.532522] [ 6992] 33 6992 87593 3228 150 0 0 apache2
> > [56793.532526] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [56793.532531] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [56793.532537] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [56793.532547] [ 7555] 33 7555 86937 1967 137 0 0 apache2
> > [56793.532553] [ 7593] 0 7593 10460 88 25 0 0 cron
> > [56793.532559] [ 7594] 0 7594 10460 88 25 0 0 cron
> > [56793.532564] [ 7595] 0 7595 10460 88 25 0 0 cron
> > [56793.532568] [ 7596] 0 7596 10460 88 25 0 0 cron
> > [56793.532573] [ 7597] 0 7597 10460 88 25 0 0 cron
> > [56793.532580] [ 7598] 0 7598 10460 88 25 0 0 cron
> > [56793.532584] [ 7599] 0 7599 10460 88 25 0 0 cron
> > [56793.532588] [ 7607] 0 7607 1002 25 8 0 0 sh
> > [56793.532593] [ 7608] 1337 7608 1002 24 8 0 0 sh
> > [56793.532597] [ 7609] 0 7609 1002 26 7 0 0 sh
> > [56793.532601] [ 7610] 0 7610 1002 24 7 0 0 sh
> > [56793.532606] [ 7611] 0 7611 1002 25 7 0 0 sh
> > [56793.532611] [ 7612] 0 7612 1002 23 8 0 0 sh
> > [56793.532615] [ 7613] 0 7613 1002 25 7 0 0 sh
> > [56793.532620] [ 7614] 0 7614 7322 156 20 0 0 wget
> > [56793.532624] [ 7615] 1337 7615 1481 19 8 0 0 sleep
> > [56793.532628] [ 7616] 0 7616 7322 110 19 0 0 wget
> > [56793.532632] [ 7617] 0 7617 7322 110 20 0 0 wget
> > [56793.532637] [ 7618] 0 7618 7322 109 19 0 0 wget
> > [56793.532641] [ 7619] 0 7619 7322 109 20 0 0 wget
> > [56793.532646] [ 7620] 0 7620 7322 109 20 0 0 wget
> > [56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or sacrifice child
> > [56793.532656] Killed process 1031 (apache2) total-vm:351516kB, anon-rss:9624kB, file-rss:9540kB
> >
> > ...
> >
> > [57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> > [57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted 2.6.50-dae-xen #2
> > [57131.763847] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff880073e21bb0
> > [57131.763858] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800788aea00
> > [57131.763867] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000004
> > [57131.763885] Call Trace:
> > [57131.763896] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> > [57131.763906] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> > [57131.763913] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> > [57131.763921] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> > [57131.763947] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> > [57131.763955] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> > [57131.763962] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> > [57131.763969] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> > [57131.763977] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> > [57131.763985] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> > [57131.763991] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> > [57131.763997] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> > [57131.764006] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> > [57131.764017] [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182
> > [57131.764023] [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7
> > [57131.764030] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> > [57131.764034] Mem-Info:
> > [57131.764037] Node 0 DMA per-cpu:
> > [57131.764041] CPU 0: hi: 0, btch: 1 usd: 0
> > [57131.764045] Node 0 DMA32 per-cpu:
> > [57131.764049] CPU 0: hi: 186, btch: 31 usd: 62
> > [57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0
> > [57131.764076] active_file:288 inactive_file:1121 isolated_file:0
> > [57131.764076] unevictable:0 dirty:0 writeback:0 unstable:0
> > [57131.764076] free:3369 slab_reclaimable:109088 slab_unreclaimable:235420
> > [57131.764076] mapped:358 shmem:8328 pagetables:821 bounce:0
> > [57131.764076] free_cma:0
> > [57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> > [57131.764115] lowmem_reserve[]: 0 1973 1973 1973
> > [57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB active_anon:46972kB inactive_anon:4104kB active_file:1152kB inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:1432kB shmem:33312kB slab_reclaimable:434592kB slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 all_unreclaimable? yes
> > [57131.764151] lowmem_reserve[]: 0 0 0 0
> > [57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
> > [57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5548kB
> > [57131.764242] 9744 total pagecache pages
> > [57131.764253] 0 pages in swap cache
> > [57131.764256] Swap cache stats: add 0, delete 0, find 0/0
> > [57131.764263] Free swap = 0kB
> > [57131.764267] Total swap = 0kB
> > [57131.770252] 524287 pages RAM
> > [57131.770260] 12090 pages reserved
> > [57131.770262] 265440 pages shared
> > [57131.770265] 433233 pages non-shared
> > [57131.770267] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> > [57131.770280] [ 330] 0 330 4213 114 12 0 -1000 udevd
> > [57131.770285] [ 815] 0 815 13048 150 20 0 0 rsyslogd
> > [57131.770290] [ 853] 104 853 12486 99 27 0 0 zabbix_agentd
> > [57131.770295] [ 909] 0 909 4212 113 12 0 -1000 udevd
> > [57131.770299] [ 910] 0 910 4212 113 12 0 -1000 udevd
> > [57131.770305] [ 1016] 0 1016 86937 2031 155 0 0 apache2
> > [57131.770310] [ 1054] 0 1054 5747 69 16 0 0 cron
> > [57131.770314] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon
> > [57131.770319] [ 1104] 106 1104 9630 147 23 0 0 ntpd
> > [57131.770323] [ 1130] 0 1130 18989 374 35 0 0 searchd
> > [57131.770328] [ 1140] 0 1140 12307 138 28 0 -1000 sshd
> > [57131.770334] [ 7363] 0 7363 10460 89 25 0 0 cron
> > [57131.770339] [ 7374] 0 7374 1002 25 7 0 0 sh
> > [57131.770343] [ 7378] 0 7378 7322 108 19 0 0 wget
> > [57131.770348] [ 8003] 0 8003 10460 88 25 0 0 cron
> > [57131.770352] [ 8006] 0 8006 10460 88 25 0 0 cron
> > [57131.770357] [ 8009] 0 8009 10460 88 25 0 0 cron
> > [57131.770361] [ 8020] 1337 8020 1002 23 7 0 0 sh
> > [57131.770366] [ 8023] 0 8023 1002 23 6 0 0 sh
> > [57131.770370] [ 8024] 0 8024 1002 24 7 0 0 sh
> > [57131.770399] [ 8030] 0 8030 7322 108 19 0 0 wget
> > [57131.770406] [ 8035] 0 8035 7322 109 20 0 0 wget
> > [57131.770410] [ 8051] 0 8051 1002 41 8 0 0 rc
> > [57131.770414] [ 8054] 0 8054 3085 60 11 0 0 startpar
> > [57131.770419] [ 8055] 1337 8055 1002 23 8 0 0 zabbix-launch-t
> > [57131.770424] [ 8057] 0 8057 1002 40 7 0 0 apache2
> > [57131.770430] [ 8067] 1337 8067 2841 76 9 0 0 apache
> > [57131.770436] [ 8084] 0 8084 1002 24 8 0 0 apache2ctl
> > [57131.770443] [ 8092] 0 8092 26561 288 48 0 0 apache2
> > [57131.770450] [ 8115] 0 8115 1002 22 8 0 0 zabbix-agent
> > [57131.770457] [ 8116] 0 8116 980 32 8 0 0 start-stop-daem
> > [57131.770461] [ 8158] 1337 8158 10130 168 22 0 0 zabbix_sender
> > [57131.770466] [ 8161] 33 8161 86937 1967 137 0 0 apache2
> > [57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or sacrifice child
> > [57131.770475] Killed process 8161 (apache2) total-vm:347748kB, anon-rss:7868kB, file-rss:0kB
> >
> >
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xen.org
> > http://lists.xen.org/xen-users
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 8:58 ` Olivier Bonvalet
@ 2013-08-07 9:35 ` Olivier Bonvalet
2013-08-07 9:46 ` Olivier Bonvalet
2013-08-07 10:16 ` Ian Campbell
0 siblings, 2 replies; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 9:35 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, xen-devel
I have a DomU which is currently running with 8GB of memory (confirmed
in /proc/meminfo), and only 6GB of memory is used, without process using
it :
root! to:~# free -m
total used free shared buffers cached
Mem: 8024 6642 1381 0 32 252
-/+ buffers/cache: 6357 1667
Swap: 255 0 255
root! to:~# ps auxw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 8408 800 ? Ss Aug06 0:01 init [2]
root 2 0.0 0.0 0 0 ? S Aug06 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Aug06 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S Aug06 0:00 [migration/0]
root 8 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_sched]
root 10 0.0 0.0 0 0 ? S Aug06 0:00 [watchdog/0]
root 11 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/1]
root 12 0.0 0.0 0 0 ? R Aug06 0:00 [migration/1]
root 13 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/1]
root 14 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/1:0]
root 15 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:0H]
root 16 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/2]
root 17 0.0 0.0 0 0 ? R Aug06 0:00 [migration/2]
root 18 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/2]
root 19 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/2:0]
root 20 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:0H]
root 21 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/3]
root 22 0.0 0.0 0 0 ? R Aug06 0:00 [migration/3]
root 23 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/3]
root 24 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/3:0]
root 25 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:0H]
root 26 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/4]
root 27 0.0 0.0 0 0 ? R Aug06 0:00 [migration/4]
root 28 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/4]
root 29 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/4:0]
root 30 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:0H]
root 31 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/5]
root 32 0.0 0.0 0 0 ? R Aug06 0:00 [migration/5]
root 33 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/5]
root 34 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/5:0]
root 35 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:0H]
root 36 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/6]
root 37 0.0 0.0 0 0 ? R Aug06 0:00 [migration/6]
root 38 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/6]
root 39 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/6:0]
root 40 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:0H]
root 41 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/7]
root 42 0.0 0.0 0 0 ? R Aug06 0:00 [migration/7]
root 43 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/7]
root 44 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/7:0]
root 45 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:0H]
root 46 0.0 0.0 0 0 ? S< Aug06 0:00 [khelper]
root 47 0.0 0.0 0 0 ? S Aug06 0:00 [kdevtmpfs]
root 48 0.0 0.0 0 0 ? S Aug06 0:00 [xenwatch]
root 49 0.0 0.0 0 0 ? S Aug06 0:00 [xenbus]
root 50 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:1H]
root 51 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:1H]
root 52 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:1H]
root 53 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:1H]
root 54 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:1H]
root 55 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:1H]
root 56 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:1H]
root 57 0.0 0.0 0 0 ? S< Aug06 0:00 [writeback]
root 58 0.0 0.0 0 0 ? S< Aug06 0:00 [kintegrityd]
root 59 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
root 60 0.0 0.0 0 0 ? S< Aug06 0:00 [kblockd]
root 61 0.0 0.0 0 0 ? S< Aug06 0:00 [xenbus_frontend]
root 62 0.0 0.0 0 0 ? S< Aug06 0:00 [ata_sff]
root 63 0.0 0.0 0 0 ? S< Aug06 0:00 [edac-poller]
root 64 0.0 0.0 0 0 ? S Aug06 0:02 [kworker/0:1]
root 106 0.0 0.0 0 0 ? S Aug06 0:00 [kswapd0]
root 107 0.0 0.0 0 0 ? SN Aug06 0:00 [ksmd]
root 108 0.0 0.0 0 0 ? S Aug06 0:00 [fsnotify_mark]
root 109 0.0 0.0 0 0 ? S< Aug06 0:00 [crypto]
root 126 0.0 0.0 0 0 ? S Aug06 0:00 [khvcd]
root 127 0.0 0.0 0 0 ? S< Aug06 0:00 [iscsi_eh]
root 128 0.0 0.0 0 0 ? S< Aug06 0:00 [deferwq]
root 129 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:1]
root 253 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvda-8]
root 254 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 257 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 259 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 520 0.0 0.0 0 0 ? S< Aug06 0:00 [kdmflush]
root 704 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdc-8]
root 706 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 707 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdb-8]
root 708 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 709 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/dm-0-8]
root 710 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 713 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
root 714 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-worker-1]
root 715 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-genwork-1]
root 716 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-submit-1]
root 717 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delalloc-]
root 718 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-fixup-1]
root 719 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-1]
root 720 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
root 721 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-rmw-1]
root 722 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-rai]
root 723 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
root 724 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-wri]
root 725 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-freespace]
root 726 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delayed-m]
root 727 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cache-1]
root 728 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-readahead]
root 729 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-flush_del]
root 730 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-qgroup-re]
root 731 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cleaner]
root 732 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-transacti]
root 1845 0.0 0.0 5984 620 tty1 Ss+ Aug06 0:00 /sbin/getty 38400 tty1
root 1846 0.0 0.0 5984 624 hvc0 Ss+ Aug06 0:00 /sbin/getty 38400 hvc0
root 1908 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:2]
root 6457 0.0 0.0 76740 3348 ? Ss 11:23 0:00 sshd: daevel [priv]
daevel 6460 0.0 0.0 76740 1928 ? S 11:23 0:00 sshd: daevel@pts/0,pts/1
daevel 6461 0.0 0.0 25040 7312 pts/0 Ss 11:23 0:00 -bash
daevel 6640 0.0 0.0 25036 7296 pts/1 Ss 11:23 0:00 -bash
root 6668 0.0 0.0 25128 1264 pts/1 S 11:23 0:00 sudo su
root 6669 0.0 0.0 45804 1348 pts/1 S 11:23 0:00 su
root 6671 0.0 0.0 19944 2168 pts/1 S 11:23 0:00 bash
daevel 8314 0.0 0.0 5924 560 pts/0 S+ 11:32 0:00 sleep 30
root 8315 0.0 0.0 16900 1176 pts/1 R+ 11:33 0:00 ps auxw
root 11919 0.0 0.0 0 0 ? S< 02:00 0:00 [kworker/0:1H]
root 11971 0.0 0.0 0 0 ? S< 02:00 0:00 [bioset]
root! to:~#
root! to:~# cat /proc/meminfo
MemTotal: 8217004 kB
MemFree: 1415044 kB
Buffers: 33128 kB
Cached: 258884 kB
SwapCached: 0 kB
Active: 123328 kB
Inactive: 182692 kB
Active(anon): 14032 kB
Inactive(anon): 344 kB
Active(file): 109296 kB
Inactive(file): 182348 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 262140 kB
SwapFree: 262140 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 14024 kB
Mapped: 4300 kB
Shmem: 396 kB
Slab: 5105244 kB
SReclaimable: 1563800 kB
SUnreclaim: 3541444 kB
KernelStack: 1360232 kB
PageTables: 860 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4370640 kB
Committed_AS: 20612 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 16248 kB
VmallocChunk: 34359721447 kB
AnonHugePages: 0 kB
DirectMap4k: 8388608 kB
DirectMap2M: 0 kB
root! to:~#
root! to:~# slabtop --once ; sleep 30 ; echo ; slabtop --once
Active / Total Objects (% used) : 17298069 / 17301644 (100.0%)
Active / Total Slabs (% used) : 597110 / 597110 (100.0%)
Active / Total Caches (% used) : 72 / 93 (77.4%)
Active / Total Size (% used) : 5045185.95K / 5045752.48K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
10290816 10290816 100% 0.25K 321588 32 2572704K kmalloc-256
1997499 1997499 100% 0.19K 95119 21 380476K dentry
1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
1305654 1305654 100% 0.19K 62174 21 248696K kmalloc-192
863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
170079 170079 100% 0.81K 4361 39 139552K task_xstate
170050 170050 100% 1.66K 8950 19 286400K task_struct
169860 169860 100% 1.06K 5662 30 181184K signal_cache
19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
19383 19383 100% 0.10K 497 39 1988K buffer_head
4608 4608 100% 0.01K 9 512 36K kmalloc-8
4592 4592 100% 0.55K 164 28 2624K radix_tree_node
4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
3660 3660 100% 0.53K 122 30 1952K inode_cache
3200 3200 100% 0.03K 25 128 100K kmalloc-32
2816 2816 100% 0.02K 11 256 44K kmalloc-16
2448 2448 100% 0.04K 24 102 96K ext4_extent_status
2112 2112 100% 0.06K 33 64 132K anon_vma
1632 1632 100% 0.04K 16 102 64K ext4_system_zone
1298 1186 91% 0.18K 59 22 236K vm_area_struct
1024 910 88% 0.50K 32 32 512K kmalloc-512
546 546 100% 0.09K 13 42 52K kmalloc-96
345 345 100% 0.34K 15 23 120K blkdev_requests
340 340 100% 0.05K 4 85 16K shared_policy_node
336 336 100% 0.19K 16 21 64K tw_sock_TCP
324 324 100% 0.11K 9 36 36K jbd2_journal_head
320 320 100% 1.00K 10 32 320K kmalloc-1024
312 312 100% 0.10K 8 39 32K flow_cache
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
195 195 100% 2.06K 13 15 416K idr_layer_cache
195 179 91% 2.06K 13 15 416K sighand_cache
192 192 100% 0.06K 3 64 12K ext4_io_end
170 170 100% 0.02K 1 170 4K numa_policy
128 128 100% 0.06K 2 64 8K kmem_cache_node
128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
125 125 100% 0.62K 5 25 80K files_cache
125 125 100% 0.16K 5 25 20K sigqueue
120 80 66% 4.00K 15 8 480K kmalloc-4096
112 112 100% 2.00K 7 16 224K kmalloc-2048
100 100 100% 0.62K 4 25 64K sock_inode_cache
96 96 100% 0.25K 3 32 24K kmem_cache
96 96 100% 0.98K 3 32 96K btrfs_inode
Active / Total Objects (% used) : 17298196 / 17301771 (100.0%)
Active / Total Slabs (% used) : 597115 / 597115 (100.0%)
Active / Total Caches (% used) : 72 / 93 (77.4%)
Active / Total Size (% used) : 5045213.76K / 5045780.30K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
10290880 10290880 100% 0.25K 321590 32 2572720K kmalloc-256
1997520 1997520 100% 0.19K 95120 21 380480K dentry
1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
1305696 1305696 100% 0.19K 62176 21 248704K kmalloc-192
863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
170079 170079 100% 0.81K 4361 39 139552K task_xstate
170050 170050 100% 1.66K 8950 19 286400K task_struct
169860 169860 100% 1.06K 5662 30 181184K signal_cache
19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
19383 19383 100% 0.10K 497 39 1988K buffer_head
4608 4608 100% 0.01K 9 512 36K kmalloc-8
4592 4592 100% 0.55K 164 28 2624K radix_tree_node
4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
3660 3660 100% 0.53K 122 30 1952K inode_cache
3200 3200 100% 0.03K 25 128 100K kmalloc-32
2816 2816 100% 0.02K 11 256 44K kmalloc-16
2448 2448 100% 0.04K 24 102 96K ext4_extent_status
2112 2112 100% 0.06K 33 64 132K anon_vma
1632 1632 100% 0.04K 16 102 64K ext4_system_zone
1298 1186 91% 0.18K 59 22 236K vm_area_struct
1024 910 88% 0.50K 32 32 512K kmalloc-512
546 546 100% 0.09K 13 42 52K kmalloc-96
345 345 100% 0.34K 15 23 120K blkdev_requests
340 340 100% 0.05K 4 85 16K shared_policy_node
336 336 100% 0.19K 16 21 64K tw_sock_TCP
324 324 100% 0.11K 9 36 36K jbd2_journal_head
320 320 100% 1.00K 10 32 320K kmalloc-1024
312 312 100% 0.10K 8 39 32K flow_cache
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
195 195 100% 2.06K 13 15 416K idr_layer_cache
195 179 91% 2.06K 13 15 416K sighand_cache
192 192 100% 0.06K 3 64 12K ext4_io_end
170 170 100% 0.02K 1 170 4K numa_policy
128 128 100% 0.06K 2 64 8K kmem_cache_node
128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
125 125 100% 0.62K 5 25 80K files_cache
125 125 100% 0.16K 5 25 20K sigqueue
120 80 66% 4.00K 15 8 480K kmalloc-4096
112 112 100% 2.00K 7 16 224K kmalloc-2048
100 100 100% 0.62K 4 25 64K sock_inode_cache
96 96 100% 0.25K 3 32 24K kmem_cache
96 96 100% 0.98K 3 32 96K btrfs_inode
root! to:~#
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 9:35 ` Olivier Bonvalet
@ 2013-08-07 9:46 ` Olivier Bonvalet
2013-08-07 10:16 ` Ian Campbell
1 sibling, 0 replies; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 9:46 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, xen-devel
Le mercredi 07 août 2013 à 11:35 +0200, Olivier Bonvalet a écrit :
> I have a DomU which is currently running with 8GB of memory (confirmed
> in /proc/meminfo), and only 6GB of memory is used, without process using
> it :
>
> root! to:~# free -m
> total used free shared buffers cached
> Mem: 8024 6642 1381 0 32 252
> -/+ buffers/cache: 6357 1667
> Swap: 255 0 255
>
>
>
> root! to:~# ps auxw
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 1 0.0 0.0 8408 800 ? Ss Aug06 0:01 init [2]
> root 2 0.0 0.0 0 0 ? S Aug06 0:00 [kthreadd]
> root 3 0.0 0.0 0 0 ? S Aug06 0:00 [ksoftirqd/0]
> root 4 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/0:0]
> root 5 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/0:0H]
> root 7 0.0 0.0 0 0 ? S Aug06 0:00 [migration/0]
> root 8 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_bh]
> root 9 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_sched]
> root 10 0.0 0.0 0 0 ? S Aug06 0:00 [watchdog/0]
> root 11 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/1]
> root 12 0.0 0.0 0 0 ? R Aug06 0:00 [migration/1]
> root 13 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/1]
> root 14 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/1:0]
> root 15 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:0H]
> root 16 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/2]
> root 17 0.0 0.0 0 0 ? R Aug06 0:00 [migration/2]
> root 18 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/2]
> root 19 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/2:0]
> root 20 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:0H]
> root 21 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/3]
> root 22 0.0 0.0 0 0 ? R Aug06 0:00 [migration/3]
> root 23 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/3]
> root 24 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/3:0]
> root 25 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:0H]
> root 26 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/4]
> root 27 0.0 0.0 0 0 ? R Aug06 0:00 [migration/4]
> root 28 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/4]
> root 29 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/4:0]
> root 30 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:0H]
> root 31 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/5]
> root 32 0.0 0.0 0 0 ? R Aug06 0:00 [migration/5]
> root 33 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/5]
> root 34 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/5:0]
> root 35 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:0H]
> root 36 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/6]
> root 37 0.0 0.0 0 0 ? R Aug06 0:00 [migration/6]
> root 38 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/6]
> root 39 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/6:0]
> root 40 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:0H]
> root 41 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/7]
> root 42 0.0 0.0 0 0 ? R Aug06 0:00 [migration/7]
> root 43 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/7]
> root 44 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/7:0]
> root 45 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:0H]
> root 46 0.0 0.0 0 0 ? S< Aug06 0:00 [khelper]
> root 47 0.0 0.0 0 0 ? S Aug06 0:00 [kdevtmpfs]
> root 48 0.0 0.0 0 0 ? S Aug06 0:00 [xenwatch]
> root 49 0.0 0.0 0 0 ? S Aug06 0:00 [xenbus]
> root 50 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:1H]
> root 51 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:1H]
> root 52 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:1H]
> root 53 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:1H]
> root 54 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:1H]
> root 55 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:1H]
> root 56 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:1H]
> root 57 0.0 0.0 0 0 ? S< Aug06 0:00 [writeback]
> root 58 0.0 0.0 0 0 ? S< Aug06 0:00 [kintegrityd]
> root 59 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
> root 60 0.0 0.0 0 0 ? S< Aug06 0:00 [kblockd]
> root 61 0.0 0.0 0 0 ? S< Aug06 0:00 [xenbus_frontend]
> root 62 0.0 0.0 0 0 ? S< Aug06 0:00 [ata_sff]
> root 63 0.0 0.0 0 0 ? S< Aug06 0:00 [edac-poller]
> root 64 0.0 0.0 0 0 ? S Aug06 0:02 [kworker/0:1]
> root 106 0.0 0.0 0 0 ? S Aug06 0:00 [kswapd0]
> root 107 0.0 0.0 0 0 ? SN Aug06 0:00 [ksmd]
> root 108 0.0 0.0 0 0 ? S Aug06 0:00 [fsnotify_mark]
> root 109 0.0 0.0 0 0 ? S< Aug06 0:00 [crypto]
> root 126 0.0 0.0 0 0 ? S Aug06 0:00 [khvcd]
> root 127 0.0 0.0 0 0 ? S< Aug06 0:00 [iscsi_eh]
> root 128 0.0 0.0 0 0 ? S< Aug06 0:00 [deferwq]
> root 129 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:1]
> root 253 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvda-8]
> root 254 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 257 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 259 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 520 0.0 0.0 0 0 ? S< Aug06 0:00 [kdmflush]
> root 704 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdc-8]
> root 706 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 707 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdb-8]
> root 708 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 709 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/dm-0-8]
> root 710 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 713 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
> root 714 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-worker-1]
> root 715 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-genwork-1]
> root 716 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-submit-1]
> root 717 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delalloc-]
> root 718 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-fixup-1]
> root 719 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-1]
> root 720 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
> root 721 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-rmw-1]
> root 722 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-rai]
> root 723 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
> root 724 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-wri]
> root 725 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-freespace]
> root 726 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delayed-m]
> root 727 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cache-1]
> root 728 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-readahead]
> root 729 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-flush_del]
> root 730 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-qgroup-re]
> root 731 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cleaner]
> root 732 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-transacti]
> root 1845 0.0 0.0 5984 620 tty1 Ss+ Aug06 0:00 /sbin/getty 38400 tty1
> root 1846 0.0 0.0 5984 624 hvc0 Ss+ Aug06 0:00 /sbin/getty 38400 hvc0
> root 1908 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:2]
> root 6457 0.0 0.0 76740 3348 ? Ss 11:23 0:00 sshd: daevel [priv]
> daevel 6460 0.0 0.0 76740 1928 ? S 11:23 0:00 sshd: daevel@pts/0,pts/1
> daevel 6461 0.0 0.0 25040 7312 pts/0 Ss 11:23 0:00 -bash
> daevel 6640 0.0 0.0 25036 7296 pts/1 Ss 11:23 0:00 -bash
> root 6668 0.0 0.0 25128 1264 pts/1 S 11:23 0:00 sudo su
> root 6669 0.0 0.0 45804 1348 pts/1 S 11:23 0:00 su
> root 6671 0.0 0.0 19944 2168 pts/1 S 11:23 0:00 bash
> daevel 8314 0.0 0.0 5924 560 pts/0 S+ 11:32 0:00 sleep 30
> root 8315 0.0 0.0 16900 1176 pts/1 R+ 11:33 0:00 ps auxw
> root 11919 0.0 0.0 0 0 ? S< 02:00 0:00 [kworker/0:1H]
> root 11971 0.0 0.0 0 0 ? S< 02:00 0:00 [bioset]
> root! to:~#
>
> root! to:~# cat /proc/meminfo
> MemTotal: 8217004 kB
> MemFree: 1415044 kB
> Buffers: 33128 kB
> Cached: 258884 kB
> SwapCached: 0 kB
> Active: 123328 kB
> Inactive: 182692 kB
> Active(anon): 14032 kB
> Inactive(anon): 344 kB
> Active(file): 109296 kB
> Inactive(file): 182348 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 262140 kB
> SwapFree: 262140 kB
> Dirty: 0 kB
> Writeback: 0 kB
> AnonPages: 14024 kB
> Mapped: 4300 kB
> Shmem: 396 kB
> Slab: 5105244 kB
> SReclaimable: 1563800 kB
> SUnreclaim: 3541444 kB
> KernelStack: 1360232 kB
> PageTables: 860 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 4370640 kB
> Committed_AS: 20612 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 16248 kB
> VmallocChunk: 34359721447 kB
> AnonHugePages: 0 kB
> DirectMap4k: 8388608 kB
> DirectMap2M: 0 kB
> root! to:~#
>
>
> root! to:~# slabtop --once ; sleep 30 ; echo ; slabtop --once
> Active / Total Objects (% used) : 17298069 / 17301644 (100.0%)
> Active / Total Slabs (% used) : 597110 / 597110 (100.0%)
> Active / Total Caches (% used) : 72 / 93 (77.4%)
> Active / Total Size (% used) : 5045185.95K / 5045752.48K (100.0%)
> Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 10290816 10290816 100% 0.25K 321588 32 2572704K kmalloc-256
> 1997499 1997499 100% 0.19K 95119 21 380476K dentry
> 1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
> 1305654 1305654 100% 0.19K 62174 21 248696K kmalloc-192
> 863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
> 180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
> 171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
> 170079 170079 100% 0.81K 4361 39 139552K task_xstate
> 170050 170050 100% 1.66K 8950 19 286400K task_struct
> 169860 169860 100% 1.06K 5662 30 181184K signal_cache
> 19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
> 19383 19383 100% 0.10K 497 39 1988K buffer_head
> 4608 4608 100% 0.01K 9 512 36K kmalloc-8
> 4592 4592 100% 0.55K 164 28 2624K radix_tree_node
> 4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
> 3660 3660 100% 0.53K 122 30 1952K inode_cache
> 3200 3200 100% 0.03K 25 128 100K kmalloc-32
> 2816 2816 100% 0.02K 11 256 44K kmalloc-16
> 2448 2448 100% 0.04K 24 102 96K ext4_extent_status
> 2112 2112 100% 0.06K 33 64 132K anon_vma
> 1632 1632 100% 0.04K 16 102 64K ext4_system_zone
> 1298 1186 91% 0.18K 59 22 236K vm_area_struct
> 1024 910 88% 0.50K 32 32 512K kmalloc-512
> 546 546 100% 0.09K 13 42 52K kmalloc-96
> 345 345 100% 0.34K 15 23 120K blkdev_requests
> 340 340 100% 0.05K 4 85 16K shared_policy_node
> 336 336 100% 0.19K 16 21 64K tw_sock_TCP
> 324 324 100% 0.11K 9 36 36K jbd2_journal_head
> 320 320 100% 1.00K 10 32 320K kmalloc-1024
> 312 312 100% 0.10K 8 39 32K flow_cache
> 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
> 232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
> 195 195 100% 2.06K 13 15 416K idr_layer_cache
> 195 179 91% 2.06K 13 15 416K sighand_cache
> 192 192 100% 0.06K 3 64 12K ext4_io_end
> 170 170 100% 0.02K 1 170 4K numa_policy
> 128 128 100% 0.06K 2 64 8K kmem_cache_node
> 128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
> 125 125 100% 0.62K 5 25 80K files_cache
> 125 125 100% 0.16K 5 25 20K sigqueue
> 120 80 66% 4.00K 15 8 480K kmalloc-4096
> 112 112 100% 2.00K 7 16 224K kmalloc-2048
> 100 100 100% 0.62K 4 25 64K sock_inode_cache
> 96 96 100% 0.25K 3 32 24K kmem_cache
> 96 96 100% 0.98K 3 32 96K btrfs_inode
>
> Active / Total Objects (% used) : 17298196 / 17301771 (100.0%)
> Active / Total Slabs (% used) : 597115 / 597115 (100.0%)
> Active / Total Caches (% used) : 72 / 93 (77.4%)
> Active / Total Size (% used) : 5045213.76K / 5045780.30K (100.0%)
> Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 10290880 10290880 100% 0.25K 321590 32 2572720K kmalloc-256
> 1997520 1997520 100% 0.19K 95120 21 380480K dentry
> 1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
> 1305696 1305696 100% 0.19K 62176 21 248704K kmalloc-192
> 863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
> 180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
> 171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
> 170079 170079 100% 0.81K 4361 39 139552K task_xstate
> 170050 170050 100% 1.66K 8950 19 286400K task_struct
> 169860 169860 100% 1.06K 5662 30 181184K signal_cache
> 19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
> 19383 19383 100% 0.10K 497 39 1988K buffer_head
> 4608 4608 100% 0.01K 9 512 36K kmalloc-8
> 4592 4592 100% 0.55K 164 28 2624K radix_tree_node
> 4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
> 3660 3660 100% 0.53K 122 30 1952K inode_cache
> 3200 3200 100% 0.03K 25 128 100K kmalloc-32
> 2816 2816 100% 0.02K 11 256 44K kmalloc-16
> 2448 2448 100% 0.04K 24 102 96K ext4_extent_status
> 2112 2112 100% 0.06K 33 64 132K anon_vma
> 1632 1632 100% 0.04K 16 102 64K ext4_system_zone
> 1298 1186 91% 0.18K 59 22 236K vm_area_struct
> 1024 910 88% 0.50K 32 32 512K kmalloc-512
> 546 546 100% 0.09K 13 42 52K kmalloc-96
> 345 345 100% 0.34K 15 23 120K blkdev_requests
> 340 340 100% 0.05K 4 85 16K shared_policy_node
> 336 336 100% 0.19K 16 21 64K tw_sock_TCP
> 324 324 100% 0.11K 9 36 36K jbd2_journal_head
> 320 320 100% 1.00K 10 32 320K kmalloc-1024
> 312 312 100% 0.10K 8 39 32K flow_cache
> 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
> 232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
> 195 195 100% 2.06K 13 15 416K idr_layer_cache
> 195 179 91% 2.06K 13 15 416K sighand_cache
> 192 192 100% 0.06K 3 64 12K ext4_io_end
> 170 170 100% 0.02K 1 170 4K numa_policy
> 128 128 100% 0.06K 2 64 8K kmem_cache_node
> 128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
> 125 125 100% 0.62K 5 25 80K files_cache
> 125 125 100% 0.16K 5 25 20K sigqueue
> 120 80 66% 4.00K 15 8 480K kmalloc-4096
> 112 112 100% 2.00K 7 16 224K kmalloc-2048
> 100 100 100% 0.62K 4 25 64K sock_inode_cache
> 96 96 100% 0.25K 3 32 24K kmem_cache
> 96 96 100% 0.98K 3 32 96K btrfs_inode
> root! to:~#
>
>
>
>
and the diff between the two slaptop :
- Active / Total Objects (% used) : 17298069 / 17301644 (100.0%)
- Active / Total Slabs (% used) : 597110 / 597110 (100.0%)
+ Active / Total Objects (% used) : 17298196 / 17301771 (100.0%)
+ Active / Total Slabs (% used) : 597115 / 597115 (100.0%)
- Active / Total Size (% used) : 5045185.95K / 5045752.48K (100.0%)
+ Active / Total Size (% used) : 5045213.76K / 5045780.30K (100.0%)
-10290816 10290816 100% 0.25K 321588 32 2572704K kmalloc-256
-1997499 1997499 100% 0.19K 95119 21 380476K dentry
+10290880 10290880 100% 0.25K 321590 32 2572720K kmalloc-256
+1997520 1997520 100% 0.19K 95120 21 380480K dentry
-1305654 1305654 100% 0.19K 62174 21 248696K kmalloc-192
+1305696 1305696 100% 0.19K 62176 21 248704K kmalloc-192
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 9:35 ` Olivier Bonvalet
2013-08-07 9:46 ` Olivier Bonvalet
@ 2013-08-07 10:16 ` Ian Campbell
2013-08-07 11:17 ` Olivier Bonvalet
1 sibling, 1 reply; 18+ messages in thread
From: Ian Campbell @ 2013-08-07 10:16 UTC (permalink / raw)
To: Olivier Bonvalet; +Cc: xen-users, xen-devel
On Wed, 2013-08-07 at 11:35 +0200, Olivier Bonvalet wrote:
> I have a DomU which is currently running with 8GB of memory (confirmed
> in /proc/meminfo), and only 6GB of memory is used, without process using
> it :
Can we see the guest cfg file and the full guest console log please.
Ian.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 10:16 ` Ian Campbell
@ 2013-08-07 11:17 ` Olivier Bonvalet
2013-08-07 13:36 ` Ian Campbell
0 siblings, 1 reply; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 11:17 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, xen-devel
Le mercredi 07 août 2013 à 11:16 +0100, Ian Campbell a écrit :
> On Wed, 2013-08-07 at 11:35 +0200, Olivier Bonvalet wrote:
> > I have a DomU which is currently running with 8GB of memory (confirmed
> > in /proc/meminfo), and only 6GB of memory is used, without process using
> > it :
>
> Can we see the guest cfg file and the full guest console log please.
>
> Ian.
>
>
Yes, I had to reboot the VM, but I can easily reproduce the problem with
a that kind of loop (the memory seem to not be released) :
daevel@to:~$ while [ 1 ] ; do free | grep Mem ; done
Mem: 8217004 1434084 6782920 0 11868 150632
Mem: 8217004 1434084 6782920 0 11868 150632
Mem: 8217004 1434140 6782864 0 11868 150632
Mem: 8217004 1434140 6782864 0 11868 150632
Mem: 8217004 1434196 6782808 0 11868 150632
Mem: 8217004 1434196 6782808 0 11868 150632
Mem: 8217004 1434252 6782752 0 11868 150632
Mem: 8217004 1434252 6782752 0 11868 150632
Mem: 8217004 1434308 6782696 0 11868 150632
Mem: 8217004 1434308 6782696 0 11868 150632
Mem: 8217004 1434308 6782696 0 11868 150632
Mem: 8217004 1434388 6782616 0 11868 150632
So, the cfg :
name = 'reto'
vcpus = 1
maxvcpus = 8
memory = 8192
vif = [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
disk = [
'/dev/rbd/sas3copies/reto-root,,xvda,w',
'/dev/rbd/sas3copies/reto-home,,xvdb,w',
'/dev/rbd/sas3copies/reto-var,,xvdc,w',
'/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
'/dev/rbd/sas3copies/reto-exim,,xvde,w',
'/dev/loop1,raw,xvdy,r',
'/dev/loop2,raw,xvdz,r' ]
kernel = '/etc/xen/kernels/reto/vmlinuz'
ramdisk = '/etc/xen/kernels/reto/initrd.img'
root = '/dev/xvda ro rootfstype=ext4'
extra = 'panic=60'
Then the console :
Parsing config from /etc/xen/reto.cfg
Daemon running with PID 20283
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
[ 0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] MPS support code is not built-in.
[ 0.000000] Using acpi=off or acpi=noirq or pci=noacpi may have problem
[ 0.000000] DMI not present or invalid.
[ 0.000000] e820: last_pfn = 0x200000 max_arch_pfn = 0x400000000
[ 0.000000] e820: last_pfn = 0x100000 max_arch_pfn = 0x400000000
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[ 0.000000] init_memory_mapping: [mem 0x1ffe00000-0x1ffffffff]
[ 0.000000] init_memory_mapping: [mem 0x1fc000000-0x1ffdfffff]
[ 0.000000] init_memory_mapping: [mem 0x180000000-0x1fbffffff]
[ 0.000000] init_memory_mapping: [mem 0x00100000-0x17fffffff]
[ 0.000000] RAMDISK: [mem 0x01bbf000-0x02541fff]
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x00000000-0x1ffffffff]
[ 0.000000] NODE_DATA [mem 0x1ff017000-0x1ff019fff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00001000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal [mem 0x100000000-0x1ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00001000-0x0009ffff]
[ 0.000000] node 0: [mem 0x00100000-0x1ffffffff]
[ 0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[ 0.000000] No local APIC present
[ 0.000000] APIC: disable apic facility
[ 0.000000] APIC: switched to apic NOOP
[ 0.000000] e820: cannot find a gap in the 32bit address range
[ 0.000000] e820: PCI devices with unassigned 32bit BARs may break!
[ 0.000000] e820: [mem 0x200100000-0x2004fffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 4.2.2 (preserve-AD)
[ 0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:8 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 27 pages/cpu @ffff8801fea00000 s78400 r8192 d24000 u262144
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2064266
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/xvda ro rootfstype=ext4 panic=60
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[ 0.000000] Memory: 8205120k/8388608k available (3405k kernel code, 388k absent, 183100k reserved, 3194k data, 736k init)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=8.
[ 0.000000] NR_IRQS:4352 nr_irqs:336 16
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] tsc: Detected 3100.028 MHz processor
[ 0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6200.05 BogoMIPS (lpj=12400112)
[ 0.004000] pid_max: default: 32768 minimum: 301
[ 0.004000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[ 0.004000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.004000] Mount-cache hash table entries: 256
[ 0.004202] Initializing cgroup subsys devices
[ 0.004274] CPU: Physical Processor ID: 0
[ 0.004280] CPU: Processor Core ID: 0
[ 0.004289] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[ 0.004289] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[ 0.004289] tlb_flushall_shift: 5
[ 0.012395] cpu 0 spinlock event irq 17
[ 0.012443] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
[ 0.012648] NMI watchdog: disabled (cpu0): hardware events not enabled
[ 0.012709] installing Xen timer for CPU 1
[ 0.012720] cpu 1 spinlock event irq 24
[ 0.012762] SMP alternatives: switching to SMP code
[ 0.020179] installing Xen timer for CPU 2
[ 0.020196] cpu 2 spinlock event irq 31
[ 0.020400] installing Xen timer for CPU 3
[ 0.020410] cpu 3 spinlock event irq 38
[ 0.020583] installing Xen timer for CPU 4
[ 0.020595] cpu 4 spinlock event irq 45
[ 0.020774] installing Xen timer for CPU 5
[ 0.020784] cpu 5 spinlock event irq 52
[ 0.020981] installing Xen timer for CPU 6
[ 0.020992] cpu 6 spinlock event irq 59
[ 0.021205] installing Xen timer for CPU 7
[ 0.021219] cpu 7 spinlock event irq 66
[ 0.021369] Brought up 8 CPUs
[ 0.021440] devtmpfs: initialized
[ 0.024352] Grant tables using version 2 layout.
[ 0.024372] Grant table initialized
[ 0.024424] NET: Registered protocol family 16
[ 0.033098] dca service started, version 1.12.1
[ 0.033624] PCI: setting up Xen PCI frontend stub
[ 0.036205] bio: create slab <bio-0> at 0
[ 0.036308] ACPI: Interpreter disabled.
[ 0.036328] xen/balloon: Initialising balloon driver.
[ 0.036328] xen-balloon: Initialising balloon driver.
[ 0.040081] vgaarb: loaded
[ 0.040146] SCSI subsystem initialized
[ 0.040256] EDAC MC: Ver: 3.0.0
[ 0.040308] PCI: System does not support PCI
[ 0.040312] PCI: System does not support PCI
[ 0.040433] Switching to clocksource xen
[ 0.040497] pnp: PnP ACPI: disabled
[ 0.041786] NET: Registered protocol family 2
[ 0.042016] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.042270] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.042388] TCP: Hash tables configured (established 65536 bind 65536)
[ 0.042420] TCP: reno registered
[ 0.042441] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[ 0.042476] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[ 0.042547] NET: Registered protocol family 1
[ 0.042586] Unpacking initramfs...
[ 0.049470] Freeing initrd memory: 9740k freed
[ 0.052127] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 0.052914] sha1_ssse3: Using AVX optimized SHA-1 implementation
[ 0.052962] sha256_ssse3: Using AVX optimized SHA-256 implementation
[ 0.052994] sha512_ssse3: Using AVX optimized SHA-512 implementation
[ 0.053303] Initialise module verification
[ 0.053315] audit: initializing netlink socket (disabled)
[ 0.053329] type=2000 audit(1375874154.717:1): initialized
[ 0.055398] msgmni has been set to 16044
[ 0.055821] Key type asymmetric registered
[ 0.055827] Asymmetric key parser 'x509' registered
[ 0.055871] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 0.055878] io scheduler noop registered (default)
[ 0.055881] io scheduler deadline registered
[ 0.055911] io scheduler cfq registered
[ 0.055965] ioatdma: Intel(R) QuickData Technology Driver 4.00
[ 0.056082] Event-channel device installed.
[ 0.206386] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 0.206624] Linux agpgart interface v0.103
[ 0.640842] Loading iSCSI transport class v2.0-870.
[ 0.640958] Initialising Xen virtual ethernet driver.
[ 0.686248] blkfront: xvda: barrier or flush: disabled using persistent grants
[ 0.690168] xvda: unknown partition table
[ 0.691395] i8042: PNP: No PS/2 controller found. Probing ports directly.
[ 1.714687] i8042: No controller found
[ 1.774442] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[ 1.774487] rtc_cmos: probe of rtc_cmos failed with error -38
[ 1.774504] xen_wdt: Xen WatchDog Timer Driver v0.01
[ 1.774545] xen_wdt: initialized (timeout=60s, nowayout=0)
[ 1.774606] device-mapper: uevent: version 1.0.3
[ 1.774659] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
[ 1.774761] Netfilter messages via NETLINK v0.30.
[ 1.774836] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 1.774941] TCP: cubic registered
[ 1.774945] Initializing XFRM netlink socket
[ 1.774951] NET: Registered protocol family 17
[ 1.774962] NET: Registered protocol family 15
[ 1.774971] Key type dns_resolver registered
[ 1.775049] Loading module verification certificates
[ 1.775717] MODSIGN: Loaded cert 'Magrathea: Glacier signing key: 6138b126c58f99141b59d11f9726480fc8670364'
[ 1.775730] registered taskstats version 1
[ 1.776382] blkfront: xvdb: barrier or flush: disabled using persistent grants
[ 1.797557] xvdb: unknown partition table
[ 1.798526] blkfront: xvdc: barrier or flush: disabled using persistent grants
[ 1.820815] xvdc: unknown partition table
[ 1.822259] blkfront: xvdd: barrier or flush: disabled using persistent grants
[ 1.853766] xvdd: unknown partition table
[ 1.854749] blkfront: xvde: barrier or flush: disabled using persistent grants
[ 1.857572] xvde: unknown partition table
[ 1.858010] Setting capacity to 2097152
[ 1.858018] xvda: detected capacity change from 0 to 1073741824
[ 1.858760] blkfront: xvdy: flush diskcache: enabled using persistent grants
[ 1.860885] xvdy: unknown partition table
[ 1.861913] blkfront: xvdz: flush diskcache: enabled using persistent grants
[ 1.863770] xvdz: unknown partition table
[ 1.864014] Setting capacity to 12582912
[ 1.864022] xvdb: detected capacity change from 0 to 6442450944
[ 1.904203] Setting capacity to 2097152
[ 1.904221] xvdc: detected capacity change from 0 to 1073741824
[ 1.904815] Setting capacity to 2097152
[ 1.904824] xvdd: detected capacity change from 0 to 1073741824
[ 1.905588] Setting capacity to 2097152
[ 1.905595] xvde: detected capacity change from 0 to 1073741824
[ 1.906034] Setting capacity to 8192
[ 1.906040] xvdy: detected capacity change from 0 to 4194304
[ 1.906344] Setting capacity to 34816
[ 1.906351] xvdz: detected capacity change from 0 to 17825792
[ 1.972186] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[ 1.972672] Freeing unused kernel memory: 736k freed
[ 1.972885] Write protecting the kernel read-only data: 6144k
[ 1.975337] Freeing unused kernel memory: 680k freed
[ 1.975809] Freeing unused kernel memory: 728k freed
Loading, please wait...
[ 2.008230] udev[147]: starting version 164
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[ 2.281423] EXT4-fs (xvda): INFO: recovery required on readonly filesystem
[ 2.281436] EXT4-fs (xvda): write access will be enabled during recovery
[ 2.789083] EXT4-fs (xvda): recovery complete
[ 2.794546] EXT4-fs (xvda): mounted filesystem with ordered data mode. Opts: (null)
Begin: Running /scripts/local-bottom ... [ 2.854763] EXT4-fs (xvdz): mounted filesystem without journal. Opts: (null)
[ 2.896036] EXT4-fs (xvdy): mounted filesystem without journal. Opts: (null)
done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Reading early boot files....
Starting the hotplug events dispatcher: udevd[ 7.844815] udev[323]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Setting up LVM Volume Groups Reading all physical volumes. This may take a while...
Found volume group "vg-reto-mysql" using metadata type lvm2
[ 8.406895] bio: create slab <bio-1> at 1
1 logical volume(s) in volume group "vg-reto-mysql" now active
.
Activating swap...done.
[ 8.478687] EXT4-fs (xvda): re-mounted. Opts: (null)
Checking root file system...fsck from util-linux-ng 2.17.2
/dev/xvda: clean, 18649/65536 files, 138314/262144 blocks
done.
[ 8.518678] EXT4-fs (xvda): re-mounted. Opts: errors=remount-ro
[ 8.561167] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Loading kernel modules...done.
Cleaning up ifupdown....
[ 8.595343] zram: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.596003] zram: Created 1 device(s) ...
Setting up networking....
Setting up swapspace version 1, size = 262140 KiB
no label, UUID=06150089-664d-4e1b-ba3a-21e10901a7d3
[ 9.610429] Adding 262140k swap on /dev/zram0. Priority:1337 extents:1 across:262140k SSFS
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
/dev/xvdc: recovering journal
/dev/xvdc: clean, 3131/65536 files, 61136/262144 blocks
/dev/xvdb: recovering journal
/dev/xvdb: clean, 44609/393216 files, 593209/1572864 blocks (check in 4 mounts)
/dev/mapper/vg--reto--mysql-data: recovering journal
/dev/mapper/vg--reto--mysql-data: clean, 334/55104 files, 17135/220160 blocks
done.
Mounting local filesystems...[ 10.716114] EXT4-fs (xvdc): mounted filesystem with ordered data mode. Opts: (null)
[ 10.870314] EXT4-fs (xvdb): mounted filesystem with ordered data mode. Opts: (null)
[ 10.905646] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: data=ordered
[ 10.952142] xor: automatically using best checksumming function:
[ 10.992068] avx : 23162.000 MB/sec
[ 11.060081] raid6: sse2x1 4307 MB/s
[ 11.128068] raid6: sse2x2 6500 MB/s
[ 11.196029] raid6: sse2x4 9198 MB/s
[ 11.196039] raid6: using algorithm sse2x4 (9198 MB/s)
[ 11.196043] raid6: using ssse3x2 recovery algorithm
[ 11.202944] bio: create slab <bio-2> at 2
[ 11.204124] Btrfs loaded
[ 11.205468] device fsid 1eaac82a-191b-4255-8899-1ca628038ceb devid 1 transid 7432 /dev/xvde
[ 11.206992] btrfs: not using ssd allocation scheme
[ 11.207001] btrfs: use lzo compression
done.
Activating swapfile swap...done.
Cleaning up temporary files....
Configuring network interfaces...done.
Cleaning up temporary files....
Reading later boot files....
Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key
failed.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Setting sysfs variables....
[ 14.427621] NET: Registered protocol family 10
[ 14.433196] ip6_tables: (C) 2000-2006 Netfilter Core Team
Starting web server: apache2Warning: DocumentRoot [/home/rodolphe_me/http/www/] does not exist
Warning: DocumentRoot [/home/rodolphe_me/http/www/] does not exist
.
Starting deferred execution scheduler: atd.
Not starting internet superserver: no services enabled.
Starting NTP server: ntpd.
Starting periodic command scheduler: cron.
[ 15.714524] sshd (1122): /proc/1122/oom_adj is deprecated, please use /proc/1122/oom_score_adj instead.
Starting OpenBSD Secure Shell server: sshd.
Starting recursive DNS server: unbound[1375874170] unbound[1121:0] warning: increased limit(open files) from 1024 to 4140
.
Starting Zabbix agent: zabbix_agentd.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..
Starting ftp server: Running: /usr/sbin/pure-ftpd -f ftp -l pam -k 95 -0 -Z -D -c 500 -Y 1 -w -O clf:/var/log/pure-ftpd/transfer.log -A -8 UTF-8 -L 5000:5 -E -H -y 5:50 -p 40000:41000 -u 1000 -I 15 -C 10 -B
Starting MTA: exim4.
Debian GNU/Linux 6.0 to hvc0
to login:
Thanks for your time.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 11:17 ` Olivier Bonvalet
@ 2013-08-07 13:36 ` Ian Campbell
2013-08-07 14:26 ` Olivier Bonvalet
2013-08-07 14:48 ` Wei Liu
0 siblings, 2 replies; 18+ messages in thread
From: Ian Campbell @ 2013-08-07 13:36 UTC (permalink / raw)
To: Olivier Bonvalet; +Cc: xen-users, Roger Pau Monne, xen-devel
On Wed, 2013-08-07 at 13:17 +0200, Olivier Bonvalet wrote:
> name = 'reto'
> vcpus = 1
> maxvcpus = 8
> memory = 8192
> vif = [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
> disk = [
> '/dev/rbd/sas3copies/reto-root,,xvda,w',
> '/dev/rbd/sas3copies/reto-home,,xvdb,w',
> '/dev/rbd/sas3copies/reto-var,,xvdc,w',
> '/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
> '/dev/rbd/sas3copies/reto-exim,,xvde,w',
>
> '/dev/loop1,raw,xvdy,r',
> '/dev/loop2,raw,xvdz,r' ]
> kernel = '/etc/xen/kernels/reto/vmlinuz'
> ramdisk = '/etc/xen/kernels/reto/initrd.img'
> root = '/dev/xvda ro rootfstype=ext4'
> extra = 'panic=60'
All looks pretty normal.
> Then the console :
>
> Parsing config from /etc/xen/reto.cfg
> Daemon running with PID 20283
> [ 0.000000] Initializing cgroup subsys cpu
> [ 0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
> [ 0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
> [ 0.000000] KERNEL supported cpus:
> [ 0.000000] Intel GenuineIntel
> [ 0.000000] ACPI in unprivileged domain disabled
> [ 0.000000] e820: BIOS-provided physical RAM map:
> [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> [ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [ 0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
This all looks good.
> [ 1.776382] blkfront: xvdb: barrier or flush: disabled using persistent grants
> [ 1.797557] xvdb: unknown partition table
> [ 1.798526] blkfront: xvdc: barrier or flush: disabled using persistent grants
> [ 1.820815] xvdc: unknown partition table
> [ 1.822259] blkfront: xvdd: barrier or flush: disabled using persistent grants
> [ 1.853766] xvdd: unknown partition table
> [ 1.854749] blkfront: xvde: barrier or flush: disabled using persistent grants
> [ 1.857572] xvde: unknown partition table
> [ 1.858010] Setting capacity to 2097152
> [ 1.858018] xvda: detected capacity change from 0 to 1073741824
> [ 1.858760] blkfront: xvdy: flush diskcache: enabled using persistent grants
> [ 1.860885] xvdy: unknown partition table
> [ 1.861913] blkfront: xvdz: flush diskcache: enabled using persistent grants
> [ 1.863770] xvdz: unknown partition table
I know persistent grants have a fixed memory overhead, not sure if that
is on the back or frontend though. Roger?
is there any way to disable persistent grants manually for testing
purposes?
> [ 8.595343] zram: module is from the staging directory, the quality is unknown, you have been warned.
> [ 8.596003] zram: Created 1 device(s) ...
My first thought was that there was a memory leak in the kernel
somewhere, but staging drivers doing "magic" things with memory make me
nervous. Can you try without zram to rule it out please?
Ian.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 13:36 ` Ian Campbell
@ 2013-08-07 14:26 ` Olivier Bonvalet
2013-08-07 21:37 ` Olivier Bonvalet
2013-08-07 14:48 ` Wei Liu
1 sibling, 1 reply; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 14:26 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, Roger Pau Monne, xen-devel
Le mercredi 07 août 2013 à 14:36 +0100, Ian Campbell a écrit :
> On Wed, 2013-08-07 at 13:17 +0200, Olivier Bonvalet wrote:
>
> > name = 'reto'
> > vcpus = 1
> > maxvcpus = 8
> > memory = 8192
> > vif = [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
> > disk = [
> > '/dev/rbd/sas3copies/reto-root,,xvda,w',
> > '/dev/rbd/sas3copies/reto-home,,xvdb,w',
> > '/dev/rbd/sas3copies/reto-var,,xvdc,w',
> > '/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
> > '/dev/rbd/sas3copies/reto-exim,,xvde,w',
> >
> > '/dev/loop1,raw,xvdy,r',
> > '/dev/loop2,raw,xvdz,r' ]
> > kernel = '/etc/xen/kernels/reto/vmlinuz'
> > ramdisk = '/etc/xen/kernels/reto/initrd.img'
> > root = '/dev/xvda ro rootfstype=ext4'
> > extra = 'panic=60'
>
> All looks pretty normal.
>
> > Then the console :
> >
> > Parsing config from /etc/xen/reto.cfg
> > Daemon running with PID 20283
> > [ 0.000000] Initializing cgroup subsys cpu
> > [ 0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
> > [ 0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
> > [ 0.000000] KERNEL supported cpus:
> > [ 0.000000] Intel GenuineIntel
> > [ 0.000000] ACPI in unprivileged domain disabled
> > [ 0.000000] e820: BIOS-provided physical RAM map:
> > [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> > [ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> > [ 0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
>
> This all looks good.
>
> > [ 1.776382] blkfront: xvdb: barrier or flush: disabled using persistent grants
> > [ 1.797557] xvdb: unknown partition table
> > [ 1.798526] blkfront: xvdc: barrier or flush: disabled using persistent grants
> > [ 1.820815] xvdc: unknown partition table
> > [ 1.822259] blkfront: xvdd: barrier or flush: disabled using persistent grants
> > [ 1.853766] xvdd: unknown partition table
> > [ 1.854749] blkfront: xvde: barrier or flush: disabled using persistent grants
> > [ 1.857572] xvde: unknown partition table
> > [ 1.858010] Setting capacity to 2097152
> > [ 1.858018] xvda: detected capacity change from 0 to 1073741824
> > [ 1.858760] blkfront: xvdy: flush diskcache: enabled using persistent grants
> > [ 1.860885] xvdy: unknown partition table
> > [ 1.861913] blkfront: xvdz: flush diskcache: enabled using persistent grants
> > [ 1.863770] xvdz: unknown partition table
>
> I know persistent grants have a fixed memory overhead, not sure if that
> is on the back or frontend though. Roger?
>
> is there any way to disable persistent grants manually for testing
> purposes?
>
> > [ 8.595343] zram: module is from the staging directory, the quality is unknown, you have been warned.
> > [ 8.596003] zram: Created 1 device(s) ...
>
> My first thought was that there was a memory leak in the kernel
> somewhere, but staging drivers doing "magic" things with memory make me
> nervous. Can you try without zram to rule it out please?
>
> Ian.
>
>
You're right, I disabled zram on this Domu (others didn't have zram).
The problem is still present.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 13:36 ` Ian Campbell
2013-08-07 14:26 ` Olivier Bonvalet
@ 2013-08-07 14:48 ` Wei Liu
1 sibling, 0 replies; 18+ messages in thread
From: Wei Liu @ 2013-08-07 14:48 UTC (permalink / raw)
To: Ian Campbell
Cc: Olivier Bonvalet, xen-users, xen-devel, wei.liu2, Roger Pau Monne
On Wed, Aug 07, 2013 at 02:36:44PM +0100, Ian Campbell wrote:
> On Wed, 2013-08-07 at 13:17 +0200, Olivier Bonvalet wrote:
>
> > name = 'reto'
> > vcpus = 1
> > maxvcpus = 8
> > memory = 8192
> > vif = [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
> > disk = [
> > '/dev/rbd/sas3copies/reto-root,,xvda,w',
> > '/dev/rbd/sas3copies/reto-home,,xvdb,w',
> > '/dev/rbd/sas3copies/reto-var,,xvdc,w',
> > '/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
> > '/dev/rbd/sas3copies/reto-exim,,xvde,w',
> >
> > '/dev/loop1,raw,xvdy,r',
> > '/dev/loop2,raw,xvdz,r' ]
> > kernel = '/etc/xen/kernels/reto/vmlinuz'
> > ramdisk = '/etc/xen/kernels/reto/initrd.img'
> > root = '/dev/xvda ro rootfstype=ext4'
> > extra = 'panic=60'
>
> All looks pretty normal.
>
> > Then the console :
> >
> > Parsing config from /etc/xen/reto.cfg
> > Daemon running with PID 20283
> > [ 0.000000] Initializing cgroup subsys cpu
> > [ 0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
> > [ 0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
> > [ 0.000000] KERNEL supported cpus:
> > [ 0.000000] Intel GenuineIntel
> > [ 0.000000] ACPI in unprivileged domain disabled
> > [ 0.000000] e820: BIOS-provided physical RAM map:
> > [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> > [ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> > [ 0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
>
> This all looks good.
>
> > [ 1.776382] blkfront: xvdb: barrier or flush: disabled using persistent grants
> > [ 1.797557] xvdb: unknown partition table
> > [ 1.798526] blkfront: xvdc: barrier or flush: disabled using persistent grants
> > [ 1.820815] xvdc: unknown partition table
> > [ 1.822259] blkfront: xvdd: barrier or flush: disabled using persistent grants
> > [ 1.853766] xvdd: unknown partition table
> > [ 1.854749] blkfront: xvde: barrier or flush: disabled using persistent grants
> > [ 1.857572] xvde: unknown partition table
> > [ 1.858010] Setting capacity to 2097152
> > [ 1.858018] xvda: detected capacity change from 0 to 1073741824
> > [ 1.858760] blkfront: xvdy: flush diskcache: enabled using persistent grants
> > [ 1.860885] xvdy: unknown partition table
> > [ 1.861913] blkfront: xvdz: flush diskcache: enabled using persistent grants
> > [ 1.863770] xvdz: unknown partition table
>
> I know persistent grants have a fixed memory overhead, not sure if that
> is on the back or frontend though. Roger?
>
> is there any way to disable persistent grants manually for testing
> purposes?
>
I don't think persistent grant can exhaust 8G of guest RAM. IIRC grant
table by default has 32K entries so persistent grant can consume 128MB
RAM at most.
Wei.
> > [ 8.595343] zram: module is from the staging directory, the quality is unknown, you have been warned.
> > [ 8.596003] zram: Created 1 device(s) ...
>
> My first thought was that there was a memory leak in the kernel
> somewhere, but staging drivers doing "magic" things with memory make me
> nervous. Can you try without zram to rule it out please?
>
> Ian.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 14:26 ` Olivier Bonvalet
@ 2013-08-07 21:37 ` Olivier Bonvalet
2013-08-08 8:58 ` Ian Campbell
2013-08-09 12:53 ` Konrad Rzeszutek Wilk
0 siblings, 2 replies; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-07 21:37 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, xen-devel, Roger Pau Monne
Le mercredi 07 août 2013 à 16:26 +0200, Olivier Bonvalet a écrit :
>
> Le mercredi 07 août 2013 à 14:36 +0100, Ian Campbell a écrit :
> > On Wed, 2013-08-07 at 13:17 +0200, Olivier Bonvalet wrote:
> >
> > > name = 'reto'
> > > vcpus = 1
> > > maxvcpus = 8
> > > memory = 8192
> > > vif = [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
> > > disk = [
> > > '/dev/rbd/sas3copies/reto-root,,xvda,w',
> > > '/dev/rbd/sas3copies/reto-home,,xvdb,w',
> > > '/dev/rbd/sas3copies/reto-var,,xvdc,w',
> > > '/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
> > > '/dev/rbd/sas3copies/reto-exim,,xvde,w',
> > >
> > > '/dev/loop1,raw,xvdy,r',
> > > '/dev/loop2,raw,xvdz,r' ]
> > > kernel = '/etc/xen/kernels/reto/vmlinuz'
> > > ramdisk = '/etc/xen/kernels/reto/initrd.img'
> > > root = '/dev/xvda ro rootfstype=ext4'
> > > extra = 'panic=60'
> >
> > All looks pretty normal.
> >
> > > Then the console :
> > >
> > > Parsing config from /etc/xen/reto.cfg
> > > Daemon running with PID 20283
> > > [ 0.000000] Initializing cgroup subsys cpu
> > > [ 0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
> > > [ 0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
> > > [ 0.000000] KERNEL supported cpus:
> > > [ 0.000000] Intel GenuineIntel
> > > [ 0.000000] ACPI in unprivileged domain disabled
> > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> > > [ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> > > [ 0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
> >
> > This all looks good.
> >
> > > [ 1.776382] blkfront: xvdb: barrier or flush: disabled using persistent grants
> > > [ 1.797557] xvdb: unknown partition table
> > > [ 1.798526] blkfront: xvdc: barrier or flush: disabled using persistent grants
> > > [ 1.820815] xvdc: unknown partition table
> > > [ 1.822259] blkfront: xvdd: barrier or flush: disabled using persistent grants
> > > [ 1.853766] xvdd: unknown partition table
> > > [ 1.854749] blkfront: xvde: barrier or flush: disabled using persistent grants
> > > [ 1.857572] xvde: unknown partition table
> > > [ 1.858010] Setting capacity to 2097152
> > > [ 1.858018] xvda: detected capacity change from 0 to 1073741824
> > > [ 1.858760] blkfront: xvdy: flush diskcache: enabled using persistent grants
> > > [ 1.860885] xvdy: unknown partition table
> > > [ 1.861913] blkfront: xvdz: flush diskcache: enabled using persistent grants
> > > [ 1.863770] xvdz: unknown partition table
> >
> > I know persistent grants have a fixed memory overhead, not sure if that
> > is on the back or frontend though. Roger?
> >
> > is there any way to disable persistent grants manually for testing
> > purposes?
> >
> > > [ 8.595343] zram: module is from the staging directory, the quality is unknown, you have been warned.
> > > [ 8.596003] zram: Created 1 device(s) ...
> >
> > My first thought was that there was a memory leak in the kernel
> > somewhere, but staging drivers doing "magic" things with memory make me
> > nervous. Can you try without zram to rule it out please?
> >
> > Ian.
> >
> >
>
> You're right, I disabled zram on this Domu (others didn't have zram).
> The problem is still present.
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
So I recompiled a kernel with the kmemleak feature. I obtain that kind
of list, but not sure that it's usefull :
unreferenced object 0xffff88007cca8a80 (size 8):
comm "swapper/0", pid 1, jiffies 4294892300 (age 150.688s)
hex dump (first 8 bytes):
74 69 6d 65 72 31 00 a5 timer1..
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810056eb>] xen_setup_timer+0x62/0xc3
[<ffffffff8134296e>] xen_cpu_up+0x5b/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c527d90 (size 16):
comm "swapper/0", pid 1, jiffies 4294892302 (age 150.680s)
hex dump (first 16 bytes):
72 65 73 63 68 65 64 31 00 6b 6b 6b 6b 6b 6b a5 resched1.kkkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8100d03e>] text_poke+0x186/0x195
[<ffffffff8134d595>] xen_smp_intr_init+0x39/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c526158 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.676s)
hex dump (first 16 bytes):
63 61 6c 6c 66 75 6e 63 31 00 6b 6b 6b 6b 6b a5 callfunc1.kkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d5db>] xen_smp_intr_init+0x7f/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007cca8bd0 (size 8):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.676s)
hex dump (first 8 bytes):
64 65 62 75 67 31 00 a5 debug1..
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d624>] xen_smp_intr_init+0xc8/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c527c38 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
63 61 6c 6c 66 75 6e 63 73 69 6e 67 6c 65 31 00 callfuncsingle1.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff812126ef>] bind_virq_to_irqhandler+0x3c/0x58
[<ffffffff8134d66d>] xen_smp_intr_init+0x111/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c5262b0 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
69 72 71 77 6f 72 6b 31 00 6b 6b 6b 6b 6b 6b a5 irqwork1.kkkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d6c6>] xen_smp_intr_init+0x16a/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007cca93b0 (size 8):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 8 bytes):
74 69 6d 65 72 32 00 a5 timer2..
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810056eb>] xen_setup_timer+0x62/0xc3
[<ffffffff8134296e>] xen_cpu_up+0x5b/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c526560 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
73 70 69 6e 6c 6f 63 6b 32 00 6b 6b 6b 6b 6b a5 spinlock2.kkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff81342cde>] xen_init_lock_cpu+0x64/0xbc
[<ffffffff81342975>] xen_cpu_up+0x62/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c527830 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
72 65 73 63 68 65 64 32 00 6b 6b 6b 6b 6b 6b a5 resched2.kkkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d595>] xen_smp_intr_init+0x39/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c5266b8 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
63 61 6c 6c 66 75 6e 63 32 00 6b 6b 6b 6b 6b a5 callfunc2.kkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d5db>] xen_smp_intr_init+0x7f/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007cca8d20 (size 8):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 8 bytes):
64 65 62 75 67 32 00 a5 debug2..
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d624>] xen_smp_intr_init+0xc8/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c5276d8 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
63 61 6c 6c 66 75 6e 63 73 69 6e 67 6c 65 32 00 callfuncsingle2.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff812126ef>] bind_virq_to_irqhandler+0x3c/0x58
[<ffffffff8134d66d>] xen_smp_intr_init+0x111/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c526810 (size 16):
comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
hex dump (first 16 bytes):
69 72 71 77 6f 72 6b 32 00 6b 6b 6b 6b 6b 6b a5 irqwork2.kkkkkk.
backtrace:
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
[<ffffffff811b7071>] kvasprintf+0x41/0x68
[<ffffffff811b70d6>] kasprintf+0x3e/0x40
[<ffffffff8134d6c6>] xen_smp_intr_init+0x16a/0x271
[<ffffffff81342c28>] xen_cpu_up+0x315/0x367
[<ffffffff81348662>] _cpu_up+0xb8/0x124
[<ffffffff81348792>] cpu_up+0xc4/0xd0
[<ffffffff8169dca6>] smp_init+0x4a/0x75
[<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffff8133d92f>] kernel_init+0x5/0xd1
[<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
[<ffffffff8133d92a>] kernel_init+0x0/0xd1
[<ffffffffffffffff>] 0xffffffffffffffff
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 21:37 ` Olivier Bonvalet
@ 2013-08-08 8:58 ` Ian Campbell
2013-08-08 10:10 ` Olivier Bonvalet
2013-08-09 12:53 ` Konrad Rzeszutek Wilk
1 sibling, 1 reply; 18+ messages in thread
From: Ian Campbell @ 2013-08-08 8:58 UTC (permalink / raw)
To: Olivier Bonvalet; +Cc: xen-users, Roger Pau Monne, xen-devel
On Wed, 2013-08-07 at 23:37 +0200, Olivier Bonvalet wrote:
> So I recompiled a kernel with the kmemleak feature. I obtain that kind
> of list, but not sure that it's usefull :
These look to me like valid things to be allocating at boot time, and
even if they are leaked there isn't enough here to exhaust 8GB by a long
way.
It'd be worth monitoring to see if it grows at all or if anything
interesting shows up after running for a while with the leak.
Likewise it'd be worth keeping an eye on the process list and slabtop
and seeing if anything appears to be growing without bound.
Other than that I'm afraid I don't have many smart ideas.
Ian.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-08 8:58 ` Ian Campbell
@ 2013-08-08 10:10 ` Olivier Bonvalet
2013-08-08 10:18 ` Ian Campbell
0 siblings, 1 reply; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-08 10:10 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, Roger Pau Monne, xen-devel
Le jeudi 08 août 2013 à 09:58 +0100, Ian Campbell a écrit :
> On Wed, 2013-08-07 at 23:37 +0200, Olivier Bonvalet wrote:
> > So I recompiled a kernel with the kmemleak feature. I obtain that kind
> > of list, but not sure that it's usefull :
>
> These look to me like valid things to be allocating at boot time, and
> even if they are leaked there isn't enough here to exhaust 8GB by a long
> way.
>
> It'd be worth monitoring to see if it grows at all or if anything
> interesting shows up after running for a while with the leak.
>
> Likewise it'd be worth keeping an eye on the process list and slabtop
> and seeing if anything appears to be growing without bound.
>
> Other than that I'm afraid I don't have many smart ideas.
>
> Ian.
>
>
Ok, then I will become crazy : when I start the kernel with kmemleak=on
in fact I haven't memory leak. The memory usage stay near 300MB.
Then I restart on the same kernel, without kmemleak=on, the memory usage
jump to 600MB and still grow.
Olivier
PS : I retry several time, to confirm that.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-08 10:10 ` Olivier Bonvalet
@ 2013-08-08 10:18 ` Ian Campbell
2013-08-08 11:43 ` Olivier Bonvalet
0 siblings, 1 reply; 18+ messages in thread
From: Ian Campbell @ 2013-08-08 10:18 UTC (permalink / raw)
To: Olivier Bonvalet; +Cc: xen-users, xen-devel, Roger Pau Monne
On Thu, 2013-08-08 at 12:10 +0200, Olivier Bonvalet wrote:
>
> Le jeudi 08 août 2013 à 09:58 +0100, Ian Campbell a écrit :
> > On Wed, 2013-08-07 at 23:37 +0200, Olivier Bonvalet wrote:
> > > So I recompiled a kernel with the kmemleak feature. I obtain that kind
> > > of list, but not sure that it's usefull :
> >
> > These look to me like valid things to be allocating at boot time, and
> > even if they are leaked there isn't enough here to exhaust 8GB by a long
> > way.
> >
> > It'd be worth monitoring to see if it grows at all or if anything
> > interesting shows up after running for a while with the leak.
> >
> > Likewise it'd be worth keeping an eye on the process list and slabtop
> > and seeing if anything appears to be growing without bound.
> >
> > Other than that I'm afraid I don't have many smart ideas.
> >
> > Ian.
> >
> >
>
> Ok, then I will become crazy : when I start the kernel with kmemleak=on
> in fact I haven't memory leak. The memory usage stay near 300MB.
>
> Then I restart on the same kernel, without kmemleak=on, the memory usage
> jump to 600MB and still grow.
>
> Olivier
>
> PS : I retry several time, to confirm that.
*boggles*
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-08 10:18 ` Ian Campbell
@ 2013-08-08 11:43 ` Olivier Bonvalet
2013-08-08 13:25 ` Wei Liu
0 siblings, 1 reply; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-08 11:43 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-users, xen-devel, Roger Pau Monne
Le jeudi 08 août 2013 à 11:18 +0100, Ian Campbell a écrit :
> On Thu, 2013-08-08 at 12:10 +0200, Olivier Bonvalet wrote:
> >
> > Le jeudi 08 août 2013 à 09:58 +0100, Ian Campbell a écrit :
> > > On Wed, 2013-08-07 at 23:37 +0200, Olivier Bonvalet wrote:
> > > > So I recompiled a kernel with the kmemleak feature. I obtain that kind
> > > > of list, but not sure that it's usefull :
> > >
> > > These look to me like valid things to be allocating at boot time, and
> > > even if they are leaked there isn't enough here to exhaust 8GB by a long
> > > way.
> > >
> > > It'd be worth monitoring to see if it grows at all or if anything
> > > interesting shows up after running for a while with the leak.
> > >
> > > Likewise it'd be worth keeping an eye on the process list and slabtop
> > > and seeing if anything appears to be growing without bound.
> > >
> > > Other than that I'm afraid I don't have many smart ideas.
> > >
> > > Ian.
> > >
> > >
> >
> > Ok, then I will become crazy : when I start the kernel with kmemleak=on
> > in fact I haven't memory leak. The memory usage stay near 300MB.
> >
> > Then I restart on the same kernel, without kmemleak=on, the memory usage
> > jump to 600MB and still grow.
> >
> > Olivier
> >
> > PS : I retry several time, to confirm that.
>
> *boggles*
>
> Ian.
>
>
So, I retried the slabtop test, with more leaked memory, to have better
visibility :
--- a 2013-08-08 12:29:48.437966407 +0200
+++ c 2013-08-08 13:33:41.213711305 +0200
@@ -1,23 +1,23 @@
- Active / Total Objects (% used) : 186382 / 189232 (98.5%)
- Active / Total Slabs (% used) : 6600 / 6600 (100.0%)
- Active / Total Caches (% used) : 100 / 151 (66.2%)
- Active / Total Size (% used) : 111474.55K / 113631.58K (98.1%)
- Minimum / Average / Maximum Object : 0.33K / 0.60K / 8.32K
+ Active / Total Objects (% used) : 2033635 / 2037851 (99.8%)
+ Active / Total Slabs (% used) : 70560 / 70560 (100.0%)
+ Active / Total Caches (% used) : 101 / 151 (66.9%)
+ Active / Total Size (% used) : 1289959.44K / 1292725.98K (99.8%)
+ Minimum / Average / Maximum Object : 0.33K / 0.63K / 8.32K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
- 55048 55038 99% 0.56K 1966 28 31456K filp
- 29536 29528 99% 0.50K 923 32 14768K cred_jar
- 22909 22909 100% 0.51K 739 31 11824K dentry
+831572 831552 99% 0.56K 29699 28 475184K filp
+501664 501635 99% 0.50K 15677 32 250832K cred_jar
+172453 172432 99% 0.51K 5563 31 89008K dentry
+150920 150906 99% 0.91K 4312 35 137984K proc_inode_cache
+ 54686 54652 99% 0.43K 1478 37 23648K task_delay_info
+ 54656 54651 99% 1.98K 3416 16 109312K task_struct
+ 54652 54651 99% 1.19K 2102 26 67264K task_xstate
+ 54648 54644 99% 0.44K 1518 36 24288K pid
+ 54648 54645 99% 1.38K 2376 23 76032K signal_cache
+ 38200 38188 99% 0.38K 1910 20 15280K kmalloc-64
11803 11774 99% 0.43K 319 37 5104K sysfs_dir_cache
- 7350 7327 99% 0.91K 210 35 6720K proc_inode_cache
- 5520 5465 99% 0.38K 276 20 2208K anon_vma_chain
- 5216 5137 98% 0.50K 163 32 2608K vm_area_struct
- 3984 3978 99% 0.33K 166 24 1328K kmalloc-8
- 3811 3798 99% 0.84K 103 37 3296K inode_cache
- 3384 3359 99% 0.44K 94 36 1504K pid
- 3381 3362 99% 1.38K 147 23 4704K signal_cache
- 3380 3366 99% 1.19K 130 26 4160K task_xstate
- 3376 3366 99% 1.98K 211 16 6752K task_struct
- 3367 3367 100% 0.43K 91 37 1456K task_delay_info
- 2886 2864 99% 0.42K 78 37 1248K buffer_head
- 2720 2714 99% 0.93K 80 34 2560K shmem_inode_cache
+ 7920 7676 96% 0.38K 396 20 3168K anon_vma_chain
+ 7808 7227 92% 0.50K 244 32 3904K vm_area_struct
+ 5624 5581 99% 0.42K 152 37 2432K buffer_head
+ 4316 4308 99% 1.22K 166 26 5312K ext4_inode_cache
+ 3984 3977 99% 0.33K 166 24 1328K kmalloc-8
So in 1 hour, "filp" and "cred_jar" eat a lot of memory.
But I have no idea what is it...
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-08 11:43 ` Olivier Bonvalet
@ 2013-08-08 13:25 ` Wei Liu
2013-08-08 14:17 ` Olivier Bonvalet
0 siblings, 1 reply; 18+ messages in thread
From: Wei Liu @ 2013-08-08 13:25 UTC (permalink / raw)
To: Olivier Bonvalet
Cc: xen-users, Roger Pau Monne, wei.liu2, Ian Campbell, xen-devel
On Thu, Aug 08, 2013 at 01:43:08PM +0200, Olivier Bonvalet wrote:
>
>
> Le jeudi 08 août 2013 à 11:18 +0100, Ian Campbell a écrit :
> > On Thu, 2013-08-08 at 12:10 +0200, Olivier Bonvalet wrote:
> > >
> > > Le jeudi 08 août 2013 à 09:58 +0100, Ian Campbell a écrit :
> > > > On Wed, 2013-08-07 at 23:37 +0200, Olivier Bonvalet wrote:
> > > > > So I recompiled a kernel with the kmemleak feature. I obtain that kind
> > > > > of list, but not sure that it's usefull :
> > > >
> > > > These look to me like valid things to be allocating at boot time, and
> > > > even if they are leaked there isn't enough here to exhaust 8GB by a long
> > > > way.
> > > >
> > > > It'd be worth monitoring to see if it grows at all or if anything
> > > > interesting shows up after running for a while with the leak.
> > > >
> > > > Likewise it'd be worth keeping an eye on the process list and slabtop
> > > > and seeing if anything appears to be growing without bound.
> > > >
> > > > Other than that I'm afraid I don't have many smart ideas.
> > > >
> > > > Ian.
> > > >
> > > >
> > >
> > > Ok, then I will become crazy : when I start the kernel with kmemleak=on
> > > in fact I haven't memory leak. The memory usage stay near 300MB.
> > >
> > > Then I restart on the same kernel, without kmemleak=on, the memory usage
> > > jump to 600MB and still grow.
> > >
> > > Olivier
> > >
> > > PS : I retry several time, to confirm that.
> >
> > *boggles*
> >
> > Ian.
> >
> >
>
>
> So, I retried the slabtop test, with more leaked memory, to have better
> visibility :
>
> --- a 2013-08-08 12:29:48.437966407 +0200
> +++ c 2013-08-08 13:33:41.213711305 +0200
> @@ -1,23 +1,23 @@
> - Active / Total Objects (% used) : 186382 / 189232 (98.5%)
> - Active / Total Slabs (% used) : 6600 / 6600 (100.0%)
> - Active / Total Caches (% used) : 100 / 151 (66.2%)
> - Active / Total Size (% used) : 111474.55K / 113631.58K (98.1%)
> - Minimum / Average / Maximum Object : 0.33K / 0.60K / 8.32K
> + Active / Total Objects (% used) : 2033635 / 2037851 (99.8%)
> + Active / Total Slabs (% used) : 70560 / 70560 (100.0%)
> + Active / Total Caches (% used) : 101 / 151 (66.9%)
> + Active / Total Size (% used) : 1289959.44K / 1292725.98K (99.8%)
> + Minimum / Average / Maximum Object : 0.33K / 0.63K / 8.32K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> - 55048 55038 99% 0.56K 1966 28 31456K filp
> - 29536 29528 99% 0.50K 923 32 14768K cred_jar
> - 22909 22909 100% 0.51K 739 31 11824K dentry
> +831572 831552 99% 0.56K 29699 28 475184K filp
> +501664 501635 99% 0.50K 15677 32 250832K cred_jar
> +172453 172432 99% 0.51K 5563 31 89008K dentry
> +150920 150906 99% 0.91K 4312 35 137984K proc_inode_cache
> + 54686 54652 99% 0.43K 1478 37 23648K task_delay_info
> + 54656 54651 99% 1.98K 3416 16 109312K task_struct
> + 54652 54651 99% 1.19K 2102 26 67264K task_xstate
> + 54648 54644 99% 0.44K 1518 36 24288K pid
> + 54648 54645 99% 1.38K 2376 23 76032K signal_cache
> + 38200 38188 99% 0.38K 1910 20 15280K kmalloc-64
> 11803 11774 99% 0.43K 319 37 5104K sysfs_dir_cache
> - 7350 7327 99% 0.91K 210 35 6720K proc_inode_cache
> - 5520 5465 99% 0.38K 276 20 2208K anon_vma_chain
> - 5216 5137 98% 0.50K 163 32 2608K vm_area_struct
> - 3984 3978 99% 0.33K 166 24 1328K kmalloc-8
> - 3811 3798 99% 0.84K 103 37 3296K inode_cache
> - 3384 3359 99% 0.44K 94 36 1504K pid
> - 3381 3362 99% 1.38K 147 23 4704K signal_cache
> - 3380 3366 99% 1.19K 130 26 4160K task_xstate
> - 3376 3366 99% 1.98K 211 16 6752K task_struct
> - 3367 3367 100% 0.43K 91 37 1456K task_delay_info
> - 2886 2864 99% 0.42K 78 37 1248K buffer_head
> - 2720 2714 99% 0.93K 80 34 2560K shmem_inode_cache
> + 7920 7676 96% 0.38K 396 20 3168K anon_vma_chain
> + 7808 7227 92% 0.50K 244 32 3904K vm_area_struct
> + 5624 5581 99% 0.42K 152 37 2432K buffer_head
> + 4316 4308 99% 1.22K 166 26 5312K ext4_inode_cache
> + 3984 3977 99% 0.33K 166 24 1328K kmalloc-8
>
> So in 1 hour, "filp" and "cred_jar" eat a lot of memory.
>
filp should be the pointer to struct file. cred_jar is slab cache to
store credentials. These two are not Xen-specific.
Dentry grows too. But that's not Xen-specific either.
Wei.
> But I have no idea what is it...
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-08 13:25 ` Wei Liu
@ 2013-08-08 14:17 ` Olivier Bonvalet
0 siblings, 0 replies; 18+ messages in thread
From: Olivier Bonvalet @ 2013-08-08 14:17 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-users, Roger Pau Monne, Ian Campbell, xen-devel
Le jeudi 08 août 2013 à 14:25 +0100, Wei Liu a écrit :
> On Thu, Aug 08, 2013 at 01:43:08PM +0200, Olivier Bonvalet wrote:
> >
> >
> > Le jeudi 08 août 2013 à 11:18 +0100, Ian Campbell a écrit :
> > > On Thu, 2013-08-08 at 12:10 +0200, Olivier Bonvalet wrote:
> > > >
> > > > Le jeudi 08 août 2013 à 09:58 +0100, Ian Campbell a écrit :
> > > > > On Wed, 2013-08-07 at 23:37 +0200, Olivier Bonvalet wrote:
> > > > > > So I recompiled a kernel with the kmemleak feature. I obtain that kind
> > > > > > of list, but not sure that it's usefull :
> > > > >
> > > > > These look to me like valid things to be allocating at boot time, and
> > > > > even if they are leaked there isn't enough here to exhaust 8GB by a long
> > > > > way.
> > > > >
> > > > > It'd be worth monitoring to see if it grows at all or if anything
> > > > > interesting shows up after running for a while with the leak.
> > > > >
> > > > > Likewise it'd be worth keeping an eye on the process list and slabtop
> > > > > and seeing if anything appears to be growing without bound.
> > > > >
> > > > > Other than that I'm afraid I don't have many smart ideas.
> > > > >
> > > > > Ian.
> > > > >
> > > > >
> > > >
> > > > Ok, then I will become crazy : when I start the kernel with kmemleak=on
> > > > in fact I haven't memory leak. The memory usage stay near 300MB.
> > > >
> > > > Then I restart on the same kernel, without kmemleak=on, the memory usage
> > > > jump to 600MB and still grow.
> > > >
> > > > Olivier
> > > >
> > > > PS : I retry several time, to confirm that.
> > >
> > > *boggles*
> > >
> > > Ian.
> > >
> > >
> >
> >
> > So, I retried the slabtop test, with more leaked memory, to have better
> > visibility :
> >
> > --- a 2013-08-08 12:29:48.437966407 +0200
> > +++ c 2013-08-08 13:33:41.213711305 +0200
> > @@ -1,23 +1,23 @@
> > - Active / Total Objects (% used) : 186382 / 189232 (98.5%)
> > - Active / Total Slabs (% used) : 6600 / 6600 (100.0%)
> > - Active / Total Caches (% used) : 100 / 151 (66.2%)
> > - Active / Total Size (% used) : 111474.55K / 113631.58K (98.1%)
> > - Minimum / Average / Maximum Object : 0.33K / 0.60K / 8.32K
> > + Active / Total Objects (% used) : 2033635 / 2037851 (99.8%)
> > + Active / Total Slabs (% used) : 70560 / 70560 (100.0%)
> > + Active / Total Caches (% used) : 101 / 151 (66.9%)
> > + Active / Total Size (% used) : 1289959.44K / 1292725.98K (99.8%)
> > + Minimum / Average / Maximum Object : 0.33K / 0.63K / 8.32K
> >
> > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> > - 55048 55038 99% 0.56K 1966 28 31456K filp
> > - 29536 29528 99% 0.50K 923 32 14768K cred_jar
> > - 22909 22909 100% 0.51K 739 31 11824K dentry
> > +831572 831552 99% 0.56K 29699 28 475184K filp
> > +501664 501635 99% 0.50K 15677 32 250832K cred_jar
> > +172453 172432 99% 0.51K 5563 31 89008K dentry
> > +150920 150906 99% 0.91K 4312 35 137984K proc_inode_cache
> > + 54686 54652 99% 0.43K 1478 37 23648K task_delay_info
> > + 54656 54651 99% 1.98K 3416 16 109312K task_struct
> > + 54652 54651 99% 1.19K 2102 26 67264K task_xstate
> > + 54648 54644 99% 0.44K 1518 36 24288K pid
> > + 54648 54645 99% 1.38K 2376 23 76032K signal_cache
> > + 38200 38188 99% 0.38K 1910 20 15280K kmalloc-64
> > 11803 11774 99% 0.43K 319 37 5104K sysfs_dir_cache
> > - 7350 7327 99% 0.91K 210 35 6720K proc_inode_cache
> > - 5520 5465 99% 0.38K 276 20 2208K anon_vma_chain
> > - 5216 5137 98% 0.50K 163 32 2608K vm_area_struct
> > - 3984 3978 99% 0.33K 166 24 1328K kmalloc-8
> > - 3811 3798 99% 0.84K 103 37 3296K inode_cache
> > - 3384 3359 99% 0.44K 94 36 1504K pid
> > - 3381 3362 99% 1.38K 147 23 4704K signal_cache
> > - 3380 3366 99% 1.19K 130 26 4160K task_xstate
> > - 3376 3366 99% 1.98K 211 16 6752K task_struct
> > - 3367 3367 100% 0.43K 91 37 1456K task_delay_info
> > - 2886 2864 99% 0.42K 78 37 1248K buffer_head
> > - 2720 2714 99% 0.93K 80 34 2560K shmem_inode_cache
> > + 7920 7676 96% 0.38K 396 20 3168K anon_vma_chain
> > + 7808 7227 92% 0.50K 244 32 3904K vm_area_struct
> > + 5624 5581 99% 0.42K 152 37 2432K buffer_head
> > + 4316 4308 99% 1.22K 166 26 5312K ext4_inode_cache
> > + 3984 3977 99% 0.33K 166 24 1328K kmalloc-8
> >
> > So in 1 hour, "filp" and "cred_jar" eat a lot of memory.
> >
>
> filp should be the pointer to struct file. cred_jar is slab cache to
> store credentials. These two are not Xen-specific.
>
> Dentry grows too. But that's not Xen-specific either.
>
>
> Wei.
>
Great. So I stop here, I will try with kernel developers.
Thanks for the help !
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Xen-users] unexpected Out Of Memory (OOM)
2013-08-07 21:37 ` Olivier Bonvalet
2013-08-08 8:58 ` Ian Campbell
@ 2013-08-09 12:53 ` Konrad Rzeszutek Wilk
1 sibling, 0 replies; 18+ messages in thread
From: Konrad Rzeszutek Wilk @ 2013-08-09 12:53 UTC (permalink / raw)
To: Olivier Bonvalet; +Cc: xen-users, Roger Pau Monne, Ian Campbell, xen-devel
> unreferenced object 0xffff88007cca8a80 (size 8):
> comm "swapper/0", pid 1, jiffies 4294892300 (age 150.688s)
> hex dump (first 8 bytes):
> 74 69 6d 65 72 31 00 a5 timer1..
> backtrace:
> [<ffffffff811b70d6>] kasprintf+0x3e/0x40
> [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
> [<ffffffff811b7071>] kvasprintf+0x41/0x68
> [<ffffffff811b70d6>] kasprintf+0x3e/0x40
> [<ffffffff810056eb>] xen_setup_timer+0x62/0xc3
> [<ffffffff8134296e>] xen_cpu_up+0x5b/0x367
> [<ffffffff81348662>] _cpu_up+0xb8/0x124
> [<ffffffff81348792>] cpu_up+0xc4/0xd0
... more of them.
Are all fixed in v3.11.
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2013-08-09 12:53 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-08-07 0:02 unexpected Out Of Memory (OOM) Olivier Bonvalet
2013-08-07 8:29 ` [Xen-users] " Ian Campbell
2013-08-07 8:58 ` Olivier Bonvalet
2013-08-07 9:35 ` Olivier Bonvalet
2013-08-07 9:46 ` Olivier Bonvalet
2013-08-07 10:16 ` Ian Campbell
2013-08-07 11:17 ` Olivier Bonvalet
2013-08-07 13:36 ` Ian Campbell
2013-08-07 14:26 ` Olivier Bonvalet
2013-08-07 21:37 ` Olivier Bonvalet
2013-08-08 8:58 ` Ian Campbell
2013-08-08 10:10 ` Olivier Bonvalet
2013-08-08 10:18 ` Ian Campbell
2013-08-08 11:43 ` Olivier Bonvalet
2013-08-08 13:25 ` Wei Liu
2013-08-08 14:17 ` Olivier Bonvalet
2013-08-09 12:53 ` Konrad Rzeszutek Wilk
2013-08-07 14:48 ` Wei Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).