Re: [patch v3] swap: virtual swap readahead

[Wu Fengguang <fengguang.wu@intel.com>]

On Tue, Jun 16, 2009 at 02:22:17AM +0800, Johannes Weiner wrote:
> On Fri, Jun 12, 2009 at 09:59:27AM +0800, Wu Fengguang wrote:
> > On Thu, Jun 11, 2009 at 06:17:42PM +0800, Johannes Weiner wrote:
> > > On Thu, Jun 11, 2009 at 01:22:28PM +0800, Wu Fengguang wrote:
> > > > Unfortunately, after fixing it up the swap readahead patch still performs slow
> > > > (even worse this time):
> > > 
> > > Thanks for doing the tests.  Do you know if the time difference comes
> > > from IO or CPU time?
> > > 
> > > Because one reason I could think of is that the original code walks
> > > the readaround window in two directions, starting from the target each
> > > time but immediately stops when it encounters a hole where the new
> > > code just skips holes but doesn't abort readaround and thus might
> > > indeed read more slots.
> > > 
> > > I have an old patch flying around that changed the physical ra code to
> > > use a bitmap that is able to represent holes.  If the increased time
> > > is waiting for IO, I would be interested if that patch has the same
> > > negative impact.
> > 
> > You can send me the patch :)
> 
> Okay, attached is a rebase against latest -mmotm.
> 
> > But for this patch it is IO bound. The CPU iowait field actually is
> > going up as the test goes on:
> 
> It's probably the larger ra window then which takes away the bandwidth
> needed to load the new executables.  This sucks.  Would be nice to
> have 'optional IO' for readahead that is dropped when normal-priority
> IO requests are coming in...  Oh, we have READA for bios.  But it
> doesn't seem to implement dropping requests on load (or I am blind).

Hi Hannes,

Sorry for the long delay! A bad news is that I get many oom with this patch:

[  781.450862] Xorg invoked oom-killer: gfp_mask=0xd2, order=0, oom_adj=0
[  781.457411] Pid: 3272, comm: Xorg Not tainted 2.6.30-rc8-mm1 #312
[  781.463511] Call Trace:
[  781.465976]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  781.471462]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  781.477210]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  781.482449]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  781.488188]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  781.493666]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  781.500015]  [<ffffffff81079ebd>] ? trace_hardirqs_on+0xd/0x10
[  781.505846]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  781.511857]  [<ffffffff810e7fe8>] __vmalloc_area_node+0xf8/0x190
[  781.517869]  [<ffffffffa014c9b5>] ? i915_gem_execbuffer+0xb45/0x12f0 [i915]
[  781.524835]  [<ffffffff810e8121>] __vmalloc_node+0xa1/0xb0
[  781.530346]  [<ffffffffa014c9b5>] ? i915_gem_execbuffer+0xb45/0x12f0 [i915]
[  781.537312]  [<ffffffffa014bf2b>] ? i915_gem_execbuffer+0xbb/0x12f0 [i915]
[  781.544192]  [<ffffffff810e8281>] vmalloc+0x21/0x30
[  781.549100]  [<ffffffffa014c9b5>] i915_gem_execbuffer+0xb45/0x12f0 [i915]
[  781.555920]  [<ffffffff81079ebd>] ? trace_hardirqs_on+0xd/0x10
[  781.561789]  [<ffffffffa00f5b7d>] drm_ioctl+0x12d/0x3d0 [drm]
[  781.567569]  [<ffffffffa014be70>] ? i915_gem_execbuffer+0x0/0x12f0 [i915]
[  781.574383]  [<ffffffff81079ebd>] ? trace_hardirqs_on+0xd/0x10
[  781.580225]  [<ffffffff8110babd>] vfs_ioctl+0x7d/0xa0
[  781.585287]  [<ffffffff8110bb6a>] do_vfs_ioctl+0x8a/0x580
[  781.590706]  [<ffffffff81078f3a>] ? lockdep_sys_exit+0x2a/0x90
[  781.596552]  [<ffffffff81544b34>] ? lockdep_sys_exit_thunk+0x35/0x67
[  781.602929]  [<ffffffff8110c0aa>] sys_ioctl+0x4a/0x80
[  781.607995]  [<ffffffff8100bf42>] system_call_fastpath+0x16/0x1b
[  781.614005] Mem-Info:
[  781.616293] Node 0 DMA per-cpu:
[  781.619471] CPU    0: hi:    0, btch:   1 usd:   0
[  781.624278] CPU    1: hi:    0, btch:   1 usd:   0
[  781.629080] Node 0 DMA32 per-cpu:
[  781.632443] CPU    0: hi:  186, btch:  31 usd:  83
[  781.637243] CPU    1: hi:  186, btch:  31 usd: 108
[  781.642045] Active_anon:41057 active_file:2334 inactive_anon:47003
[  781.642048]  inactive_file:2148 unevictable:4 dirty:0 writeback:0 unstable:0
[  781.642051]  free:1180 slab:14177 mapped:4473 pagetables:7629 bounce:0
[  781.661802] Node 0 DMA free:2016kB min:84kB low:104kB high:124kB active_anon:5408kB inactive_anon:5676kB active_file:16kB inactive_file:0kB unevictable:0kB present:15164kB pages_scanned:42276 all_unreclaimable? no
[  781.680773] lowmem_reserve[]: 0 483 483 483
[  781.685089] Node 0 DMA32 free:2704kB min:2768kB low:3460kB high:4152kB active_anon:158820kB inactive_anon:182224kB active_file:9320kB inactive_file:8592kB unevictable:16kB present:495008kB pages_scanned:673623 all_unreclaimable? yes
[  781.705711] lowmem_reserve[]: 0 0 0 0
[  781.709501] Node 0 DMA: 104*4kB 0*8kB 6*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2016kB
[  781.720553] Node 0 DMA32: 318*4kB 1*8kB 1*16kB 6*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2704kB
[  781.731764] 61569 total pagecache pages
[  781.735618] 6489 pages in swap cache
[  781.739212] Swap cache stats: add 285146, delete 278657, find 31455/133061
[  781.746092] Free swap  = 709316kB
[  781.749417] Total swap = 1048568kB
[  781.759726] 131072 pages RAM
[  781.762645] 9628 pages reserved
[  781.765793] 95620 pages shared
[  781.768862] 58466 pages non-shared
[  781.772278] Out of memory: kill process 3487 (run-many-x-apps) score 1471069 or a child
[  781.780291] Killed process 3488 (xeyes)
[  781.830240] gtali invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
[  781.837208] Pid: 4113, comm: gtali Not tainted 2.6.30-rc8-mm1 #312
[  781.843554] Call Trace:
[  781.846233]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  781.851870]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  781.857615]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  781.862840]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  781.868578]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  781.874054]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  781.880401]  [<ffffffff810f3fb6>] alloc_page_vma+0x86/0x1c0
[  781.885969]  [<ffffffff810e9d08>] read_swap_cache_async+0xd8/0x120
[  781.892147]  [<ffffffff810e9f05>] swapin_readahead+0xb5/0x110
[  781.897886]  [<ffffffff810dac73>] do_swap_page+0x403/0x510
[  781.903366]  [<ffffffff810e9933>] ? lookup_swap_cache+0x13/0x30
[  781.909279]  [<ffffffff810da8ea>] ? do_swap_page+0x7a/0x510
[  781.914850]  [<ffffffff810dc72e>] handle_mm_fault+0x44e/0x500
[  781.920587]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  781.926149]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  781.931287] Mem-Info:
[  781.933559] Node 0 DMA per-cpu:
[  781.936714] CPU    0: hi:    0, btch:   1 usd:   0
[  781.941500] CPU    1: hi:    0, btch:   1 usd:   0
[  781.946288] Node 0 DMA32 per-cpu:
[  781.949615] CPU    0: hi:  186, btch:  31 usd:  84
[  781.954402] CPU    1: hi:  186, btch:  31 usd: 109
[  781.959192] Active_anon:41029 active_file:2334 inactive_anon:46908
[  781.959193]  inactive_file:2211 unevictable:4 dirty:0 writeback:0 unstable:0
[  781.959194]  free:1180 slab:14177 mapped:4492 pagetables:7608 bounce:0
[  781.978897] Node 0 DMA free:2016kB min:84kB low:104kB high:124kB active_anon:5296kB inactive_anon:5408kB active_file:16kB inactive_file:176kB unevictable:0kB present:15164kB pages_scanned:6816 all_unreclaimable? no
[  781.997900] lowmem_reserve[]: 0 483 483 483
[  782.002173] Node 0 DMA32 free:2704kB min:2768kB low:3460kB high:4152kB active_anon:158820kB inactive_anon:182224kB active_file:9320kB inactive_file:8668kB unevictable:16kB present:495008kB pages_scanned:674199 all_unreclaimable? yes
[  782.022740] lowmem_reserve[]: 0 0 0 0
[  782.026488] Node 0 DMA: 82*4kB 9*8kB 7*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2016kB
[  782.037309] Node 0 DMA32: 318*4kB 1*8kB 1*16kB 6*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2704kB
[  782.048405] 61637 total pagecache pages
[  782.052236] 6494 pages in swap cache
[  782.055809] Swap cache stats: add 285154, delete 278660, find 31456/133069
[  782.062672] Free swap  = 709592kB
[  782.065983] Total swap = 1048568kB
[  782.072735] 131072 pages RAM
[  782.075632] 9628 pages reserved
[  782.078774] 95669 pages shared
[  782.081822] 58413 pages non-shared
[  782.085223] Out of memory: kill process 3487 (run-many-x-apps) score 1466556 or a child
[  782.093215] Killed process 3566 (gthumb)
[  790.063897] gnome-panel invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[  790.071664] Pid: 3405, comm: gnome-panel Not tainted 2.6.30-rc8-mm1 #312
[  790.078421] Call Trace:
[  790.080902]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  790.086410]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  790.092159]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  790.097387]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  790.103135]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  790.108632]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  790.115001]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  790.121002]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[  790.126745]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[  790.133352]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[  790.140057]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[  790.145103]  [<ffffffff810c1497>] filemap_fault+0x3f7/0x400
[  790.150678]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[  790.155902]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[  790.161989]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[  790.167738]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  790.173304]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  790.178441] Mem-Info:
[  790.180714] Node 0 DMA per-cpu:
[  790.183870] CPU    0: hi:    0, btch:   1 usd:   0
[  790.188659] CPU    1: hi:    0, btch:   1 usd:   0
[  790.193446] Node 0 DMA32 per-cpu:
[  790.196783] CPU    0: hi:  186, btch:  31 usd:  43
[  790.201569] CPU    1: hi:  186, btch:  31 usd:  31
[  790.206359] Active_anon:41179 active_file:900 inactive_anon:46967
[  790.206360]  inactive_file:4104 unevictable:4 dirty:0 writeback:0 unstable:0
[  790.206361]  free:1165 slab:13961 mapped:3241 pagetables:7475 bounce:0
[  790.225984] Node 0 DMA free:2012kB min:84kB low:104kB high:124kB active_anon:5496kB inactive_anon:5800kB active_file:4kB inactive_file:220kB unevictable:0kB present:15164kB pages_scanned:26112 all_unreclaimable? yes
[  790.245079] lowmem_reserve[]: 0 483 483 483
[  790.249352] Node 0 DMA32 free:2648kB min:2768kB low:3460kB high:4152kB active_anon:159220kB inactive_anon:182068kB active_file:3596kB inactive_file:16196kB unevictable:16kB present:495008kB pages_scanned:875456 all_unreclaimable? yes
[  790.270005] lowmem_reserve[]: 0 0 0 0
[  790.273762] Node 0 DMA: 53*4kB 9*8kB 12*16kB 2*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2012kB
[  790.284681] Node 0 DMA32: 190*4kB 46*8kB 7*16kB 6*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2648kB
[  790.295866] 62097 total pagecache pages
[  790.299698] 6548 pages in swap cache
[  790.303271] Swap cache stats: add 286032, delete 279484, find 31565/133879
[  790.310137] Free swap  = 717460kB
[  790.313445] Total swap = 1048568kB
[  790.320544] 131072 pages RAM
[  790.323445] 9628 pages reserved
[  790.326591] 85371 pages shared
[  790.329641] 59742 pages non-shared
[  790.333046] Out of memory: kill process 3487 (run-many-x-apps) score 1258333 or a child
[  790.341039] Killed process 3599 (gedit)
[  790.382081] gedit used greatest stack depth: 2064 bytes left
[  792.149572] Xorg invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0
[  792.156786] Pid: 3272, comm: Xorg Not tainted 2.6.30-rc8-mm1 #312
[  792.162980] Call Trace:
[  792.165429]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  792.170937]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  792.176691]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  792.181909]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  792.187653]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  792.193136]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  792.199490]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  792.205491]  [<ffffffff810c7409>] __get_free_pages+0x9/0x50
[  792.211060]  [<ffffffff8110e402>] __pollwait+0xc2/0x100
[  792.216283]  [<ffffffff81495903>] unix_poll+0x23/0xc0
[  792.221330]  [<ffffffff81419ac8>] sock_poll+0x18/0x20
[  792.226380]  [<ffffffff8110d9a9>] do_select+0x3e9/0x730
[  792.231597]  [<ffffffff8110d5c0>] ? do_select+0x0/0x730
[  792.236816]  [<ffffffff8110e340>] ? __pollwait+0x0/0x100
[  792.242126]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.247180]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.252227]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.257275]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.262331]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.267377]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.272422]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.277468]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.282519]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[  792.287574]  [<ffffffff8110deef>] core_sys_select+0x1ff/0x330
[  792.293317]  [<ffffffff8110dd38>] ? core_sys_select+0x48/0x330
[  792.299162]  [<ffffffffa014954c>] ? i915_gem_throttle_ioctl+0x4c/0x60 [i915]
[  792.306204]  [<ffffffff81079ebd>] ? trace_hardirqs_on+0xd/0x10
[  792.312034]  [<ffffffff810706cc>] ? getnstimeofday+0x5c/0xf0
[  792.317687]  [<ffffffff8106acb9>] ? ktime_get_ts+0x59/0x60
[  792.323169]  [<ffffffff8110e27a>] sys_select+0x4a/0x110
[  792.328387]  [<ffffffff8100bf42>] system_call_fastpath+0x16/0x1b
[  792.334389] Mem-Info:
[  792.336663] Node 0 DMA per-cpu:
[  792.339824] CPU    0: hi:    0, btch:   1 usd:   0
[  792.344612] CPU    1: hi:    0, btch:   1 usd:   0
[  792.349397] Node 0 DMA32 per-cpu:
[  792.352734] CPU    0: hi:  186, btch:  31 usd:  57
[  792.357518] CPU    1: hi:  186, btch:  31 usd:  50
[  792.362310] Active_anon:40862 active_file:1622 inactive_anon:47020
[  792.362311]  inactive_file:3746 unevictable:4 dirty:0 writeback:0 unstable:0
[  792.362313]  free:1187 slab:13902 mapped:4052 pagetables:7387 bounce:0
[  792.382030] Node 0 DMA free:2012kB min:84kB low:104kB high:124kB active_anon:5428kB inactive_anon:5680kB active_file:0kB inactive_file:224kB unevictable:0kB present:15164kB pages_scanned:4992 all_unreclaimable? no
[  792.400957] lowmem_reserve[]: 0 483 483 483
[  792.405232] Node 0 DMA32 free:2736kB min:2768kB low:3460kB high:4152kB active_anon:158020kB inactive_anon:182284kB active_file:6488kB inactive_file:14760kB unevictable:16kB present:495008kB pages_scanned:876741 all_unreclaimable? yes
[  792.425889] lowmem_reserve[]: 0 0 0 0
[  792.429637] Node 0 DMA: 31*4kB 14*8kB 15*16kB 2*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2012kB
[  792.440651] Node 0 DMA32: 86*4kB 95*8kB 14*16kB 6*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2736kB
[  792.451821] 62288 total pagecache pages
[  792.455655] 6442 pages in swap cache
[  792.459230] Swap cache stats: add 286223, delete 279781, find 31574/134040
[  792.466100] Free swap  = 723520kB
[  792.469405] Total swap = 1048568kB
[  792.476461] 131072 pages RAM
[  792.479359] 9628 pages reserved
[  792.482502] 86274 pages shared
[  792.485547] 59031 pages non-shared
[  792.488956] Out of memory: kill process 3487 (run-many-x-apps) score 1235901 or a child
[  792.496952] Killed process 3626 (xpdf.bin)
[  912.097890] gnome-control-c invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[  912.105967] Pid: 5395, comm: gnome-control-c Not tainted 2.6.30-rc8-mm1 #312
[  912.113042] Call Trace:
[  912.115499]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  912.120994]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  912.126737]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  912.131961]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  912.137709]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  912.143193]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  912.149547]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  912.155551]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[  912.161295]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[  912.167904]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[  912.174602]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[  912.179650]  [<ffffffff810c1497>] filemap_fault+0x3f7/0x400
[  912.185221]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[  912.190445]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[  912.196539]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[  912.202278]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  912.207840]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  912.212976] Mem-Info:
[  912.215247] Node 0 DMA per-cpu:
[  912.218402] CPU    0: hi:    0, btch:   1 usd:   0
[  912.223190] CPU    1: hi:    0, btch:   1 usd:   0
[  912.227979] Node 0 DMA32 per-cpu:
[  912.231315] CPU    0: hi:  186, btch:  31 usd: 118
[  912.236100] CPU    1: hi:  186, btch:  31 usd: 158
[  912.240891] Active_anon:42350 active_file:809 inactive_anon:47098
[  912.240892]  inactive_file:2682 unevictable:4 dirty:0 writeback:3 unstable:0
[  912.240893]  free:1164 slab:13886 mapped:3078 pagetables:7561 bounce:0
[  912.260546] Node 0 DMA free:2016kB min:84kB low:104kB high:124kB active_anon:5456kB inactive_anon:5676kB active_file:4kB inactive_file:72kB unevictable:0kB present:15164kB pages_scanned:1920 all_unreclaimable? no
[  912.279403] lowmem_reserve[]: 0 483 483 483
[  912.283671] Node 0 DMA32 free:2600kB min:2768kB low:3460kB high:4152kB active_anon:163944kB inactive_anon:182600kB active_file:3232kB inactive_file:10644kB unevictable:16kB present:495008kB pages_scanned:571360 all_unreclaimable? yes
[  912.304335] lowmem_reserve[]: 0 0 0 0
[  912.308082] Node 0 DMA: 22*4kB 16*8kB 12*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2008kB
[  912.319093] Node 0 DMA32: 128*4kB 131*8kB 1*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2600kB
[  912.330367] 62393 total pagecache pages
[  912.334201] 7186 pages in swap cache
[  912.337778] Swap cache stats: add 320003, delete 312817, find 34852/153688
[  912.344648] Free swap  = 714408kB
[  912.347950] Total swap = 1048568kB
[  912.355114] 131072 pages RAM
[  912.358011] 9628 pages reserved
[  912.361153] 84608 pages shared
[  912.364199] 58138 pages non-shared
[  912.367606] Out of memory: kill process 3487 (run-many-x-apps) score 1281073 or a child
[  912.375604] Killed process 3669 (xterm)
[  912.427936] tty_ldisc_deref: no references.
[  912.480847] nautilus invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[  912.487981] Pid: 3408, comm: nautilus Not tainted 2.6.30-rc8-mm1 #312
[  912.494418] Call Trace:
[  912.496876]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  912.502361]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  912.508100]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  912.513327]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  912.519067]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  912.524552]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  912.530902]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  912.536907]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[  912.542645]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[  912.549253]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[  912.555946]  [<ffffffff810a9c9b>] ? delayacct_end+0x6b/0xa0
[  912.561517]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[  912.566563]  [<ffffffff810cacb3>] ondemand_readahead+0x163/0x2d0
[  912.572563]  [<ffffffff810caf25>] page_cache_sync_readahead+0x25/0x30
[  912.579000]  [<ffffffff810c141c>] filemap_fault+0x37c/0x400
[  912.584576]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[  912.589799]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[  912.595888]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[  912.601632]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  912.607206]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  912.612345] Mem-Info:
[  912.614624] Node 0 DMA per-cpu:
[  912.617787] CPU    0: hi:    0, btch:   1 usd:   0
[  912.622570] CPU    1: hi:    0, btch:   1 usd:   0
[  912.627353] Node 0 DMA32 per-cpu:
[  912.630682] CPU    0: hi:  186, btch:  31 usd: 121
[  912.635470] CPU    1: hi:  186, btch:  31 usd:  76
[  912.640259] Active_anon:42310 active_file:830 inactive_anon:47085
[  912.640260]  inactive_file:2747 unevictable:4 dirty:0 writeback:0 unstable:0
[  912.640261]  free:1182 slab:13881 mapped:3111 pagetables:7523 bounce:0
[  912.659881] Node 0 DMA free:2004kB min:84kB low:104kB high:124kB active_anon:5468kB inactive_anon:5784kB active_file:4kB inactive_file:56kB unevictable:0kB present:15164kB pages_scanned:5152 all_unreclaimable? no
[  912.678724] lowmem_reserve[]: 0 483 483 483
[  912.682990] Node 0 DMA32 free:2724kB min:2768kB low:3460kB high:4152kB active_anon:163772kB inactive_anon:182556kB active_file:3316kB inactive_file:10932kB unevictable:16kB present:495008kB pages_scanned:51712 all_unreclaimable? no
[  912.703478] lowmem_reserve[]: 0 0 0 0
[  912.707226] Node 0 DMA: 21*4kB 16*8kB 12*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB
[  912.718239] Node 0 DMA32: 159*4kB 132*8kB 1*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2732kB
[  912.729502] 62461 total pagecache pages
[  912.733337] 7171 pages in swap cache
[  912.736915] Swap cache stats: add 320011, delete 312840, find 34852/153696
[  912.743782] Free swap  = 715668kB
[  912.747098] Total swap = 1048568kB
[  912.754168] 131072 pages RAM
[  912.757059] 9628 pages reserved
[  912.760191] 84519 pages shared
[  912.763248] 58139 pages non-shared
[  912.766653] Out of memory: kill process 3487 (run-many-x-apps) score 1273781 or a child
[  912.774647] Killed process 3762 (gnome-terminal)
[  913.650490] tty_ldisc_deref: no references.
[  914.671325] kerneloops-appl invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
[  914.679083] Pid: 3425, comm: kerneloops-appl Not tainted 2.6.30-rc8-mm1 #312
[  914.686121] Call Trace:
[  914.688575]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  914.694057]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  914.699800]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  914.705034]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  914.710791]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  914.716279]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  914.722640]  [<ffffffff810f3fb6>] alloc_page_vma+0x86/0x1c0
[  914.728208]  [<ffffffff810e9d08>] read_swap_cache_async+0xd8/0x120
[  914.734391]  [<ffffffff810e9f05>] swapin_readahead+0xb5/0x110
[  914.740139]  [<ffffffff810dac73>] do_swap_page+0x403/0x510
[  914.745632]  [<ffffffff810c0710>] ? find_get_page+0x0/0x110
[  914.751200]  [<ffffffff810e9933>] ? lookup_swap_cache+0x13/0x30
[  914.757115]  [<ffffffff810da8ea>] ? do_swap_page+0x7a/0x510
[  914.762688]  [<ffffffff810dc72e>] handle_mm_fault+0x44e/0x500
[  914.768437]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  914.774005]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  914.779136] Mem-Info:
[  914.781410] Node 0 DMA per-cpu:
[  914.784572] CPU    0: hi:    0, btch:   1 usd:   0
[  914.789367] CPU    1: hi:    0, btch:   1 usd:   0
[  914.794156] Node 0 DMA32 per-cpu:
[  914.797493] CPU    0: hi:  186, btch:  31 usd: 150
[  914.802278] CPU    1: hi:  186, btch:  31 usd: 147
[  914.807064] Active_anon:42324 active_file:1285 inactive_anon:47097
[  914.807065]  inactive_file:2225 unevictable:4 dirty:0 writeback:0 unstable:0
[  914.807067]  free:1185 slab:13908 mapped:3648 pagetables:7413 bounce:0
[  914.826781] Node 0 DMA free:2016kB min:84kB low:104kB high:124kB active_anon:5360kB inactive_anon:5784kB active_file:0kB inactive_file:0kB unevictable:0kB present:15164kB pages_scanned:17408 all_unreclaimable? yes
[  914.845718] lowmem_reserve[]: 0 483 483 483
[  914.849988] Node 0 DMA32 free:2724kB min:2768kB low:3460kB high:4152kB active_anon:163936kB inactive_anon:182604kB active_file:5140kB inactive_file:8908kB unevictable:16kB present:495008kB pages_scanned:581760 all_unreclaimable? yes
[  914.870559] lowmem_reserve[]: 0 0 0 0
[  914.874306] Node 0 DMA: 37*4kB 10*8kB 12*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2020kB
[  914.885318] Node 0 DMA32: 119*4kB 139*8kB 7*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2724kB
[  914.896588] 62441 total pagecache pages
[  914.900417] 7199 pages in swap cache
[  914.903999] Swap cache stats: add 320272, delete 313073, find 34864/153895
[  914.910867] Free swap  = 721224kB
[  914.914193] Total swap = 1048568kB
[  914.921489] 131072 pages RAM
[  914.924370] 9628 pages reserved
[  914.927519] 84507 pages shared
[  914.930581] 57535 pages non-shared
[  914.933989] Out of memory: kill process 3487 (run-many-x-apps) score 1213315 or a child
[  914.941986] Killed process 3803 (urxvt)
[  914.947298] tty_ldisc_deref: no references.
[  919.983335] gnome-keyboard- invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
[  919.991145] Pid: 5458, comm: gnome-keyboard- Not tainted 2.6.30-rc8-mm1 #312
[  919.998198] Call Trace:
[  920.000663]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  920.006157]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  920.011906]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  920.017135]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  920.022876]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  920.028357]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  920.034706]  [<ffffffff810f3fb6>] alloc_page_vma+0x86/0x1c0
[  920.040280]  [<ffffffff810e9d08>] read_swap_cache_async+0xd8/0x120
[  920.046460]  [<ffffffff810e9f05>] swapin_readahead+0xb5/0x110
[  920.052196]  [<ffffffff810dac73>] do_swap_page+0x403/0x510
[  920.057676]  [<ffffffff810e9933>] ? lookup_swap_cache+0x13/0x30
[  920.063592]  [<ffffffff810da8ea>] ? do_swap_page+0x7a/0x510
[  920.069165]  [<ffffffff810dc72e>] handle_mm_fault+0x44e/0x500
[  920.074901]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  920.080470]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  920.085604] Mem-Info:
[  920.087875] Node 0 DMA per-cpu:
[  920.091031] CPU    0: hi:    0, btch:   1 usd:   0
[  920.095818] CPU    1: hi:    0, btch:   1 usd:   0
[  920.100617] Node 0 DMA32 per-cpu:
[  920.103947] CPU    0: hi:  186, btch:  31 usd:  89
[  920.108734] CPU    1: hi:  186, btch:  31 usd: 119
[  920.113524] Active_anon:42944 active_file:542 inactive_anon:46956
[  920.113525]  inactive_file:2652 unevictable:4 dirty:0 writeback:0 unstable:0
[  920.113526]  free:1169 slab:13893 mapped:3036 pagetables:7342 bounce:0
[  920.133149] Node 0 DMA free:2008kB min:84kB low:104kB high:124kB active_anon:5568kB inactive_anon:5772kB active_file:20kB inactive_file:164kB unevictable:0kB present:15164kB pages_scanned:22824 all_unreclaimable? yes
[  920.152324] lowmem_reserve[]: 0 483 483 483
[  920.156597] Node 0 DMA32 free:2668kB min:2768kB low:3460kB high:4152kB active_anon:166208kB inactive_anon:182052kB active_file:2148kB inactive_file:10444kB unevictable:16kB present:495008kB pages_scanned:650400 all_unreclaimable? yes
[  920.177245] lowmem_reserve[]: 0 0 0 0
[  920.180991] Node 0 DMA: 44*4kB 9*8kB 10*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2008kB
[  920.191903] Node 0 DMA32: 165*4kB 117*8kB 3*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2668kB
[  920.203169] 62409 total pagecache pages
[  920.207000] 7469 pages in swap cache
[  920.210572] Swap cache stats: add 321003, delete 313534, find 34989/154507
[  920.217436] Free swap  = 725812kB
[  920.220752] Total swap = 1048568kB
[  920.227856] 131072 pages RAM
[  920.230752] 9628 pages reserved
[  920.233901] 78560 pages shared
[  920.236958] 58011 pages non-shared
[  920.240355] Out of memory: kill process 3487 (run-many-x-apps) score 1195965 or a child
[  920.248346] Killed process 3889 (gnome-system-mo)
[  920.993872] nautilus invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[  921.001843] Pid: 3408, comm: nautilus Not tainted 2.6.30-rc8-mm1 #312
[  921.008294] Call Trace:
[  921.010757]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  921.016245]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  921.021995]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  921.027215]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  921.032954]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  921.038441]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  921.044805]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  921.050808]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[  921.056549]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[  921.063163]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[  921.069868]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[  921.074918]  [<ffffffff810c1497>] filemap_fault+0x3f7/0x400
[  921.080487]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[  921.085717]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[  921.091805]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[  921.097552]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  921.103145]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  921.108280] Mem-Info:
[  921.110556] Node 0 DMA per-cpu:
[  921.113720] CPU    0: hi:    0, btch:   1 usd:   0
[  921.118501] CPU    1: hi:    0, btch:   1 usd:   0
[  921.123286] Node 0 DMA32 per-cpu:
[  921.126614] CPU    0: hi:  186, btch:  31 usd:  25
[  921.131400] CPU    1: hi:  186, btch:  31 usd:  58
[  921.136187] Active_anon:42277 active_file:992 inactive_anon:46953
[  921.136188]  inactive_file:3279 unevictable:4 dirty:0 writeback:0 unstable:0
[  921.136189]  free:1183 slab:13728 mapped:3449 pagetables:7235 bounce:0
[  921.155810] Node 0 DMA free:2016kB min:84kB low:104kB high:124kB active_anon:5540kB inactive_anon:5772kB active_file:20kB inactive_file:224kB unevictable:0kB present:15164kB pages_scanned:18464 all_unreclaimable? yes
[  921.174995] lowmem_reserve[]: 0 483 483 483
[  921.179259] Node 0 DMA32 free:2716kB min:2768kB low:3460kB high:4152kB active_anon:163568kB inactive_anon:182040kB active_file:3948kB inactive_file:12892kB unevictable:16kB present:495008kB pages_scanned:719674 all_unreclaimable? yes
[  921.199914] lowmem_reserve[]: 0 0 0 0
[  921.203661] Node 0 DMA: 50*4kB 7*8kB 10*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2016kB
[  921.214577] Node 0 DMA32: 257*4kB 45*8kB 19*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2716kB
[  921.225837] 63208 total pagecache pages
[  921.229675] 7214 pages in swap cache
[  921.233249] Swap cache stats: add 321070, delete 313856, find 34991/154562
[  921.240112] Free swap  = 730844kB
[  921.243427] Total swap = 1048568kB
[  921.250566] 131072 pages RAM
[  921.253460] 9628 pages reserved
[  921.256599] 79050 pages shared
[  921.259646] 57895 pages non-shared
[  921.263048] Out of memory: kill process 3487 (run-many-x-apps) score 1168892 or a child
[  921.271042] Killed process 3917 (gnome-help)
[  934.057490] firefox-bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[  934.065285] Pid: 3353, comm: firefox-bin Not tainted 2.6.30-rc8-mm1 #312
[  934.072425] Call Trace:
[  934.074882]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  934.080382]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  934.086126]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  934.091349]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  934.097091]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  934.102568]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  934.108914]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  934.114922]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[  934.120667]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[  934.127269]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[  934.133963]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[  934.139018]  [<ffffffff810c1497>] filemap_fault+0x3f7/0x400
[  934.144593]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[  934.149812]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[  934.155898]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[  934.161640]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  934.167208]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  934.172348] Mem-Info:
[  934.174614] Node 0 DMA per-cpu:
[  934.177775] CPU    0: hi:    0, btch:   1 usd:   0
[  934.182560] CPU    1: hi:    0, btch:   1 usd:   0
[  934.187342] Node 0 DMA32 per-cpu:
[  934.190671] CPU    0: hi:  186, btch:  31 usd: 115
[  934.195459] CPU    1: hi:  186, btch:  31 usd: 146
[  934.200251] Active_anon:43024 active_file:1381 inactive_anon:46959
[  934.200252]  inactive_file:2292 unevictable:4 dirty:0 writeback:0 unstable:0
[  934.200253]  free:1170 slab:13755 mapped:4121 pagetables:7012 bounce:0
[  934.219958] Node 0 DMA free:2000kB min:84kB low:104kB high:124kB active_anon:5532kB inactive_anon:5756kB active_file:16kB inactive_file:248kB unevictable:0kB present:15164kB pages_scanned:18348 all_unreclaimable? yes
[  934.239142] lowmem_reserve[]: 0 483 483 483
[  934.243408] Node 0 DMA32 free:2680kB min:2768kB low:3460kB high:4152kB active_anon:166564kB inactive_anon:182080kB active_file:5508kB inactive_file:8920kB unevictable:16kB present:495008kB pages_scanned:689667 all_unreclaimable? yes
[  934.263988] lowmem_reserve[]: 0 0 0 0
[  934.267735] Node 0 DMA: 60*4kB 0*8kB 10*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
[  934.278662] Node 0 DMA32: 294*4kB 2*8kB 9*16kB 10*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2680kB
[  934.289834] 62846 total pagecache pages
[  934.293669] 7202 pages in swap cache
[  934.297244] Swap cache stats: add 322861, delete 315659, find 35288/156117
[  934.304107] Free swap  = 758748kB
[  934.307422] Total swap = 1048568kB
[  934.314470] 131072 pages RAM
[  934.317362] 9628 pages reserved
[  934.320501] 76930 pages shared
[  934.323549] 57149 pages non-shared
[  934.326955] Out of memory: kill process 3487 (run-many-x-apps) score 1006662 or a child
[  934.334948] Killed process 3952 (gnome-dictionar)
[  934.340708] firefox-bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[  934.348622] Pid: 3353, comm: firefox-bin Not tainted 2.6.30-rc8-mm1 #312
[  934.355318] Call Trace:
[  934.357768]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[  934.363256]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[  934.368998]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[  934.372992]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[  934.372992]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[  934.385506]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[  934.389481]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[  934.397856]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[  934.401848]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[  934.410200]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[  934.416894]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[  934.421942]  [<ffffffff810c1497>] filemap_fault+0x3f7/0x400
[  934.425936]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[  934.432734]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[  934.438822]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[  934.444566]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[  934.448558]  [<ffffffff81545a95>] page_fault+0x25/0x30
[  934.455262] Mem-Info:
[  934.457533] Node 0 DMA per-cpu:
[  934.460695] CPU    0: hi:    0, btch:   1 usd:   0
[  934.464690] CPU    1: hi:    0, btch:   1 usd:   0
[  934.470263] Node 0 DMA32 per-cpu:
[  934.473589] CPU    0: hi:  186, btch:  31 usd: 172
[  934.478377] CPU    1: hi:  186, btch:  31 usd: 145
[  934.482373] Active_anon:42768 active_file:1390 inactive_anon:46967
[  934.482373]  inactive_file:2301 unevictable:4 dirty:0 writeback:0 unstable:0
[  934.482373]  free:1495 slab:13778 mapped:4137 pagetables:6916 bounce:0
[  934.502869] Node 0 DMA free:2060kB min:84kB low:104kB high:124kB active_anon:5492kB inactive_anon:5788kB active_file:28kB inactive_file:252kB unevictable:0kB present:15164kB pages_scanned:0 all_unreclaimable? no
[  934.521612] lowmem_reserve[]: 0 483 483 483
[  934.525885] Node 0 DMA32 free:3920kB min:2768kB low:3460kB high:4152kB active_anon:165580kB inactive_anon:182080kB active_file:5532kB inactive_file:8952kB unevictable:16kB present:495008kB pages_scanned:0 all_unreclaimable? no
[  934.545927] lowmem_reserve[]: 0 0 0 0
[  934.549677] Node 0 DMA: 71*4kB 2*8kB 10*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2060kB
[  934.560588] Node 0 DMA32: 588*4kB 10*8kB 9*16kB 10*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 3920kB
[  934.568475] 62739 total pagecache pages
[  934.575685] 7086 pages in swap cache
[  934.579254] Swap cache stats: add 322861, delete 315775, find 35288/156117
[  934.586118] Free swap  = 763384kB
[  934.589433] Total swap = 1048568kB
[  934.597155] 131072 pages RAM
[  934.600036] 9628 pages reserved
[  934.600235] 76640 pages shared
[  934.606236] 56884 pages non-shared
[  934.609634] Out of memory: kill process 3487 (run-many-x-apps) score 978701 or a child
[  934.617540] Killed process 4014 (sol)
[ 1028.279307] firefox-bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[ 1028.286714] Pid: 5554, comm: firefox-bin Not tainted 2.6.30-rc8-mm1 #312
[ 1028.293414] Call Trace:
[ 1028.295874]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[ 1028.301361]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[ 1028.307109]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[ 1028.312330]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[ 1028.318069]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[ 1028.323554]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[ 1028.329900]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[ 1028.335899]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[ 1028.341639]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[ 1028.348247]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[ 1028.354935]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[ 1028.359982]  [<ffffffff810cacb3>] ondemand_readahead+0x163/0x2d0
[ 1028.365986]  [<ffffffff810caf25>] page_cache_sync_readahead+0x25/0x30
[ 1028.372422]  [<ffffffff810c141c>] filemap_fault+0x37c/0x400
[ 1028.377985]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[ 1028.383205]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[ 1028.389291]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[ 1028.395031]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[ 1028.400594]  [<ffffffff81545a95>] page_fault+0x25/0x30
[ 1028.405726] Mem-Info:
[ 1028.408001] Node 0 DMA per-cpu:
[ 1028.411161] CPU    0: hi:    0, btch:   1 usd:   0
[ 1028.416012] CPU    1: hi:    0, btch:   1 usd:   0
[ 1028.420860] Node 0 DMA32 per-cpu:
[ 1028.424346] CPU    0: hi:  186, btch:  31 usd: 125
[ 1028.429129] CPU    1: hi:  186, btch:  31 usd:  17
[ 1028.433914] Active_anon:41222 active_file:1015 inactive_anon:47978
[ 1028.433915]  inactive_file:4149 unevictable:4 dirty:0 writeback:0 unstable:0
[ 1028.433916]  free:1168 slab:13459 mapped:4432 pagetables:6766 bounce:0
[ 1028.453622] Node 0 DMA free:2000kB min:84kB low:104kB high:124kB active_anon:5520kB inactive_anon:5776kB active_file:0kB inactive_file:84kB unevictable:0kB present:15164kB pages_scanned:16704 all_unreclaimable? no
[ 1028.472548] lowmem_reserve[]: 0 483 483 483
[ 1028.476811] Node 0 DMA32 free:2672kB min:2768kB low:3460kB high:4152kB active_anon:159368kB inactive_anon:186136kB active_file:4060kB inactive_file:16512kB unevictable:16kB present:495008kB pages_scanned:566633 all_unreclaimable? yes
[ 1028.497459] lowmem_reserve[]: 0 0 0 0
[ 1028.501203] Node 0 DMA: 56*4kB 0*8kB 11*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
[ 1028.512136] Node 0 DMA32: 278*4kB 3*8kB 4*16kB 8*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2672kB
[ 1028.523222] 64013 total pagecache pages
[ 1028.527049] 6900 pages in swap cache
[ 1028.530627] Swap cache stats: add 334539, delete 327639, find 36253/163064
[ 1028.537490] Free swap  = 775384kB
[ 1028.540803] Total swap = 1048568kB
[ 1028.547522] 131072 pages RAM
[ 1028.550399] 9628 pages reserved
[ 1028.553550] 79539 pages shared
[ 1028.556607] 57450 pages non-shared
[ 1028.560008] Out of memory: kill process 3487 (run-many-x-apps) score 938661 or a child
[ 1028.567914] Killed process 4046 (gnometris)
[ 1162.209886] Xorg invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0
[ 1162.216441] Pid: 3272, comm: Xorg Not tainted 2.6.30-rc8-mm1 #312
[ 1162.222536] Call Trace:
[ 1162.224993]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[ 1162.230485]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[ 1162.236231]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[ 1162.241461]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[ 1162.247198]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[ 1162.252677]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[ 1162.259027]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[ 1162.265027]  [<ffffffff810c7409>] __get_free_pages+0x9/0x50
[ 1162.270599]  [<ffffffff8110e402>] __pollwait+0xc2/0x100
[ 1162.275815]  [<ffffffff81495903>] unix_poll+0x23/0xc0
[ 1162.280860]  [<ffffffff81419ac8>] sock_poll+0x18/0x20
[ 1162.285907]  [<ffffffff8110d9a9>] do_select+0x3e9/0x730
[ 1162.291129]  [<ffffffff8110d5c0>] ? do_select+0x0/0x730
[ 1162.296349]  [<ffffffff8110e340>] ? __pollwait+0x0/0x100
[ 1162.301659]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.306706]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.311748]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.316792]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.321840]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.326886]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.331933]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.336979]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.342029]  [<ffffffff8110e440>] ? pollwake+0x0/0x60
[ 1162.347071]  [<ffffffff8110deef>] core_sys_select+0x1ff/0x330
[ 1162.352807]  [<ffffffff8110dd38>] ? core_sys_select+0x48/0x330
[ 1162.358644]  [<ffffffffa014954c>] ? i915_gem_throttle_ioctl+0x4c/0x60 [i915]
[ 1162.365687]  [<ffffffff81079ebd>] ? trace_hardirqs_on+0xd/0x10
[ 1162.371511]  [<ffffffff810706cc>] ? getnstimeofday+0x5c/0xf0
[ 1162.377161]  [<ffffffff8106acb9>] ? ktime_get_ts+0x59/0x60
[ 1162.382641]  [<ffffffff8110e27a>] sys_select+0x4a/0x110
[ 1162.387863]  [<ffffffff8100bf42>] system_call_fastpath+0x16/0x1b
[ 1162.393865] Mem-Info:
[ 1162.396132] Node 0 DMA per-cpu:
[ 1162.399294] CPU    0: hi:    0, btch:   1 usd:   0
[ 1162.404076] CPU    1: hi:    0, btch:   1 usd:   0
[ 1162.408858] Node 0 DMA32 per-cpu:
[ 1162.412185] CPU    0: hi:  186, btch:  31 usd: 161
[ 1162.416972] CPU    1: hi:  186, btch:  31 usd: 182
[ 1162.421762] Active_anon:42731 active_file:740 inactive_anon:48110
[ 1162.421763]  inactive_file:2851 unevictable:4 dirty:0 writeback:0 unstable:0
[ 1162.421764]  free:1174 slab:13321 mapped:3702 pagetables:6595 bounce:0
[ 1162.441384] Node 0 DMA free:2008kB min:84kB low:104kB high:124kB active_anon:5552kB inactive_anon:5812kB active_file:0kB inactive_file:0kB unevictable:0kB present:15164kB pages_scanned:9376 all_unreclaimable? no
[ 1162.460128] lowmem_reserve[]: 0 483 483 483
[ 1162.464392] Node 0 DMA32 free:2688kB min:2768kB low:3460kB high:4152kB active_anon:165372kB inactive_anon:186628kB active_file:2960kB inactive_file:11404kB unevictable:16kB present:495008kB pages_scanned:675382 all_unreclaimable? yes
[ 1162.485048] lowmem_reserve[]: 0 0 0 0
[ 1162.488797] Node 0 DMA: 56*4kB 1*8kB 11*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2008kB
[ 1162.499720] Node 0 DMA32: 274*4kB 3*8kB 8*16kB 7*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2688kB
[ 1162.510803] 62374 total pagecache pages
[ 1162.514635] 6690 pages in swap cache
[ 1162.518210] Swap cache stats: add 344648, delete 337958, find 37585/169560
[ 1162.525071] Free swap  = 796012kB
[ 1162.528385] Total swap = 1048568kB
[ 1162.535461] 131072 pages RAM
[ 1162.538352] 9628 pages reserved
[ 1162.541490] 73953 pages shared
[ 1162.544536] 58149 pages non-shared
[ 1162.547940] Out of memory: kill process 3487 (run-many-x-apps) score 918444 or a child
[ 1162.555846] Killed process 4079 (gnect)
[ 1162.634031] /usr/games/gnom invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[ 1162.641791] Pid: 4259, comm: /usr/games/gnom Not tainted 2.6.30-rc8-mm1 #312
[ 1162.648843] Call Trace:
[ 1162.651302]  [<ffffffff81545006>] ? _spin_unlock+0x26/0x30
[ 1162.656786]  [<ffffffff810c37cc>] oom_kill_process+0xdc/0x270
[ 1162.662531]  [<ffffffff810c3b2f>] ? badness+0x18f/0x300
[ 1162.667761]  [<ffffffff810c3dd5>] __out_of_memory+0x135/0x170
[ 1162.673511]  [<ffffffff810c3f05>] out_of_memory+0xf5/0x180
[ 1162.678995]  [<ffffffff810c857c>] __alloc_pages_nodemask+0x6ac/0x6c0
[ 1162.685345]  [<ffffffff810f3ea8>] alloc_pages_current+0x78/0x100
[ 1162.691347]  [<ffffffff810c0c7b>] __page_cache_alloc+0xb/0x10
[ 1162.697086]  [<ffffffff810ca910>] __do_page_cache_readahead+0x120/0x240
[ 1162.703701]  [<ffffffff810ca8a2>] ? __do_page_cache_readahead+0xb2/0x240
[ 1162.710401]  [<ffffffff810caa4c>] ra_submit+0x1c/0x20
[ 1162.715446]  [<ffffffff810c1497>] filemap_fault+0x3f7/0x400
[ 1162.721012]  [<ffffffff810d9893>] __do_fault+0x53/0x510
[ 1162.726236]  [<ffffffff81271ce0>] ? __down_read_trylock+0x20/0x60
[ 1162.732333]  [<ffffffff810dc4a9>] handle_mm_fault+0x1c9/0x500
[ 1162.738088]  [<ffffffff81548274>] do_page_fault+0x1c4/0x330
[ 1162.743659]  [<ffffffff81545a95>] page_fault+0x25/0x30
[ 1162.748793] Mem-Info:
[ 1162.751069] Node 0 DMA per-cpu:
[ 1162.754231] CPU    0: hi:    0, btch:   1 usd:   0
[ 1162.759021] CPU    1: hi:    0, btch:   1 usd:   0
[ 1162.763812] Node 0 DMA32 per-cpu:
[ 1162.767147] CPU    0: hi:  186, btch:  31 usd:  90
[ 1162.771930] CPU    1: hi:  186, btch:  31 usd:  89
[ 1162.776719] Active_anon:42484 active_file:760 inactive_anon:48078
[ 1162.776721]  inactive_file:3351 unevictable:4 dirty:0 writeback:0 unstable:0
[ 1162.776722]  free:1174 slab:13329 mapped:3807 pagetables:6487 bounce:0
[ 1162.796351] Node 0 DMA free:2008kB min:84kB low:104kB high:124kB active_anon:5532kB inactive_anon:5812kB active_file:4kB inactive_file:0kB unevictable:0kB present:15164kB pages_scanned:1408 all_unreclaimable? no
[ 1162.815110] lowmem_reserve[]: 0 483 483 483
[ 1162.819378] Node 0 DMA32 free:2688kB min:2768kB low:3460kB high:4152kB active_anon:164404kB inactive_anon:186500kB active_file:3036kB inactive_file:13404kB unevictable:16kB present:495008kB pages_scanned:40768 all_unreclaimable? no
[ 1162.839863] lowmem_reserve[]: 0 0 0 0
[ 1162.843612] Node 0 DMA: 57*4kB 1*8kB 11*16kB 2*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2012kB
[ 1162.854539] Node 0 DMA32: 274*4kB 4*8kB 8*16kB 7*32kB 1*64kB 3*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2696kB
[ 1162.865631] 62784 total pagecache pages
[ 1162.869465] 6595 pages in swap cache
[ 1162.873034] Swap cache stats: add 344648, delete 338053, find 37585/169561
[ 1162.879901] Free swap  = 802992kB
[ 1162.883222] Total swap = 1048568kB
[ 1162.891314] 131072 pages RAM
[ 1162.894216] 9628 pages reserved
[ 1162.897365] 74036 pages shared
[ 1162.900414] 58276 pages non-shared
[ 1162.903825] Out of memory: kill process 3487 (run-many-x-apps) score 890891 or a child
[ 1162.911747] Killed process 4113 (gtali)


Thanks,
Fengguang