Re: oom_killer crash linux system

linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed

* Re: oom_killer crash linux system
       [not found]     ` <1287454058.2078.12.camel@myhost>
@ 2010-10-19  2:59       ` KAMEZAWA Hiroyuki
  2010-10-19  5:23         ` Minchan Kim
  0 siblings, 1 reply; 11+ messages in thread
From: KAMEZAWA Hiroyuki @ 2010-10-19  2:59 UTC (permalink / raw)
  To: Figo.zhang
  Cc: lKOSAKI Motohiro, Wu Fengguang, linux-kernel@vger.kernel.org,
	rientjes@google.com, figo1802, linux-mm@kvack.org

On Tue, 19 Oct 2010 10:07:38 +0800
"Figo.zhang" <zhangtianfei@leadcoretech.com> wrote:

> 
> > 
> > very lots of change ;)
> > can you please send us your crash log?
> 
> i add some prink in select_bad_process() and oom_badness() to see
> pid/totalpages/points/memoryuseage/and finally process to selet to kill.
> 
> i found it the oom-killer select: syslog-ng,mysqld,nautilus,VirtualBox
> to kill, so my question is:
> 
> 1. the syslog-ng,mysqld,nautilus is the system foundamental process, so
> if oom-killer kill those process, the system will be damaged, such as
> lose some important data.
> 
> 2. the new oom-killer just use percentage of used memory as score to
> select the candidate to kill, but how to know this process to very
> important for system?
> 

The kernel can never know it. Just an admin (a man or management software) knows.
Old kernel tries to guess it, but it tend to be wrong and many many report comes
"why my ....is killed..." All guesswork the kernel does is not enough, I think.

> oom_score_adj, it is anyone commercial linux distributions to use this
> to protect the critical process.
> 
oom_adj may be used in some system. All my customers select panic_at_oom=1
and cause cluster fail over rather than half-broken.

<Off topic>
Your another choice is memory cgroup, I think.
please see documentation/cgroup/memory.txt or libcgroup.
http://sourceforge.net/projects/libcg/
You can use some fancy controls with it.
</Off topic>


BTW, there seems to be some strange things.
(CC'ed to linux-mm)
Brief Summary:
   an oom-killer happens on swapless environment with 2.6.36-rc8.
   It has 2G memory.
a reporter says
==
> i want to test the oom-killer. My desktop (Dell optiplex 780, i686
> kernel)have 2GB ram, i turn off the swap partition, and open a huge pdf
> files and applications, and let the system eat huge ram.
> 
> in 2.6.35, i can use ram up to 1.75GB,
> 
> but in 2.6.36-rc8, i just use to 1.53GB ram , the system come very slow
> and crashed after some minutes , the DiskIO is very busy. i see the
> DiskIO read is up to 8MB/s, write just only 400KB/s, (see by conky).
==

The trigger of oom-kill is order=0 allocation. (see original mail for full log)


Oct 19 09:44:08 myhost kernel: [  618.441470] httpd invoked oom-killer:
gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Zone's stat is.

Oct 19 09:44:08 myhost kernel: [  618.441551]
DMA free:7968kB min:64kB low:80kB high:96kB active_anon:3700kB inactive_anon:3752kB
    active_file:12kB inactive_file:252kB unevictable:0kB isolated(anon):0kB
    isolated(file):0kB present:15788kB mlocked:0kB dirty:0kB writeback:4kB
    mapped:52kB shmem:348kB slab_reclaimable:0kB slab_unreclaimable:16kB
    kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
    writeback_tmp:0kB pages_scanned:421 all_unreclaimable? yes
    lowmem_reserve[]: 0 865 1980 1980

Oct 19 09:44:08 myhost kernel: [  618.441560]
Normal free:39348kB min:3728kB low:4660kB high:5592kB active_anon:176740kB
       inactive_anon:25640kB active_file:84kB inactive_file:308kB
       unevictable:0kB isolated(anon):0kB isolated(file):0kB present:885944kB
       mlocked:0kB dirty:0kB writeback:4kB mapped:576992kB shmem:5024kB
       slab_reclaimable:7612kB slab_unreclaimable:15512kB kernel_stack:2792kB
       pagetables:6884kB unstable:0kB bounce:0kB writeback_tmp:0kB
       pages_scanned:741 all_unreclaimable? yes
       lowmem_reserve[]: 0 0 8921 8921

Oct 19 09:44:08 myhost kernel: [  618.441569]
HighMem free:392kB min:512kB low:1712kB high:2912kB active_anon:492208kB
        inactive_anon:166404kB active_file:180kB inactive_file:840kB
        unevictable:40kB isolated(anon):0kB isolated(file):0kB present:1141984kB
        mlocked:40kB dirty:0kB writeback:12kB mapped:493648kB shmem:72216kB
        slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
        pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
        pages_scanned:1552 all_unreclaimable? yes

Highmem seems a bit strange.
  present(1141984) - active_anon - inactive_anon - inactive_file - active_file
  = 482352kB but free is 392kB.

  Highmem is used for some other purpose than usual user's page.(pagetable is 0.)
  And, Hmm, mapped:493648kB seems too large for me. 
  (active/inactive-file + shmem is not enough.)
  And "mapped" in NORMAL zone is large, too.

  Does anyone have idea about file-mapped-but-not-on-LRU pages ?

Thanks,
-Kame
  





--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-19  2:59       ` oom_killer crash linux system KAMEZAWA Hiroyuki
@ 2010-10-19  5:23         ` Minchan Kim
  2010-10-19  5:26           ` KAMEZAWA Hiroyuki
  0 siblings, 1 reply; 11+ messages in thread
From: Minchan Kim @ 2010-10-19  5:23 UTC (permalink / raw)
  To: KAMEZAWA Hiroyuki
  Cc: Figo.zhang, lKOSAKI Motohiro, Wu Fengguang,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

On Tue, Oct 19, 2010 at 11:59 AM, KAMEZAWA Hiroyuki
<kamezawa.hiroyu@jp.fujitsu.com> wrote:
> On Tue, 19 Oct 2010 10:07:38 +0800
> "Figo.zhang" <zhangtianfei@leadcoretech.com> wrote:
>
>>
>> >
>> > very lots of change ;)
>> > can you please send us your crash log?
>>
>> i add some prink in select_bad_process() and oom_badness() to see
>> pid/totalpages/points/memoryuseage/and finally process to selet to kill.
>>
>> i found it the oom-killer select: syslog-ng,mysqld,nautilus,VirtualBox
>> to kill, so my question is:
>>
>> 1. the syslog-ng,mysqld,nautilus is the system foundamental process, so
>> if oom-killer kill those process, the system will be damaged, such as
>> lose some important data.
>>
>> 2. the new oom-killer just use percentage of used memory as score to
>> select the candidate to kill, but how to know this process to very
>> important for system?
>>
>
> The kernel can never know it. Just an admin (a man or management software) knows.
> Old kernel tries to guess it, but it tend to be wrong and many many report comes
> "why my ....is killed..." All guesswork the kernel does is not enough, I think.
>
>> oom_score_adj, it is anyone commercial linux distributions to use this
>> to protect the critical process.
>>
> oom_adj may be used in some system. All my customers select panic_at_oom=1
> and cause cluster fail over rather than half-broken.
>
> <Off topic>
> Your another choice is memory cgroup, I think.
> please see documentation/cgroup/memory.txt or libcgroup.
> http://sourceforge.net/projects/libcg/
> You can use some fancy controls with it.
> </Off topic>
>
>
> BTW, there seems to be some strange things.
> (CC'ed to linux-mm)
> Brief Summary:
>   an oom-killer happens on swapless environment with 2.6.36-rc8.
>   It has 2G memory.
> a reporter says
> ==
>> i want to test the oom-killer. My desktop (Dell optiplex 780, i686
>> kernel)have 2GB ram, i turn off the swap partition, and open a huge pdf
>> files and applications, and let the system eat huge ram.
>>
>> in 2.6.35, i can use ram up to 1.75GB,
>>
>> but in 2.6.36-rc8, i just use to 1.53GB ram , the system come very slow
>> and crashed after some minutes , the DiskIO is very busy. i see the
>> DiskIO read is up to 8MB/s, write just only 400KB/s, (see by conky).
> ==
>
> The trigger of oom-kill is order=0 allocation. (see original mail for full log)
>
>
> Oct 19 09:44:08 myhost kernel: [  618.441470] httpd invoked oom-killer:
> gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
>
> Zone's stat is.
>
> Oct 19 09:44:08 myhost kernel: [  618.441551]
> DMA free:7968kB min:64kB low:80kB high:96kB active_anon:3700kB inactive_anon:3752kB
>    active_file:12kB inactive_file:252kB unevictable:0kB isolated(anon):0kB
>    isolated(file):0kB present:15788kB mlocked:0kB dirty:0kB writeback:4kB
>    mapped:52kB shmem:348kB slab_reclaimable:0kB slab_unreclaimable:16kB
>    kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
>    writeback_tmp:0kB pages_scanned:421 all_unreclaimable? yes
>    lowmem_reserve[]: 0 865 1980 1980
>
> Oct 19 09:44:08 myhost kernel: [  618.441560]
> Normal free:39348kB min:3728kB low:4660kB high:5592kB active_anon:176740kB
>       inactive_anon:25640kB active_file:84kB inactive_file:308kB
>       unevictable:0kB isolated(anon):0kB isolated(file):0kB present:885944kB
>       mlocked:0kB dirty:0kB writeback:4kB mapped:576992kB shmem:5024kB
>       slab_reclaimable:7612kB slab_unreclaimable:15512kB kernel_stack:2792kB
>       pagetables:6884kB unstable:0kB bounce:0kB writeback_tmp:0kB
>       pages_scanned:741 all_unreclaimable? yes
>       lowmem_reserve[]: 0 0 8921 8921
>
> Oct 19 09:44:08 myhost kernel: [  618.441569]
> HighMem free:392kB min:512kB low:1712kB high:2912kB active_anon:492208kB
>        inactive_anon:166404kB active_file:180kB inactive_file:840kB
>        unevictable:40kB isolated(anon):0kB isolated(file):0kB present:1141984kB
>        mlocked:40kB dirty:0kB writeback:12kB mapped:493648kB shmem:72216kB
>        slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
>        pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
>        pages_scanned:1552 all_unreclaimable? yes
>
> Highmem seems a bit strange.
>  present(1141984) - active_anon - inactive_anon - inactive_file - active_file
>  = 482352kB but free is 392kB.
>
>  Highmem is used for some other purpose than usual user's page.(pagetable is 0.)
>  And, Hmm, mapped:493648kB seems too large for me.
>  (active/inactive-file + shmem is not enough.)
>  And "mapped" in NORMAL zone is large, too.
>
>  Does anyone have idea about file-mapped-but-not-on-LRU pages ?

Isn't it possible some file pages are much sharable?
Please see the page_add_file_rmap.

>
> Thanks,
> -Kame
>
>
>
>
>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
>



-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-19  5:23         ` Minchan Kim
@ 2010-10-19  5:26           ` KAMEZAWA Hiroyuki
  2010-10-19  5:34             ` Minchan Kim
  2010-10-20  1:35             ` Wu Fengguang
  0 siblings, 2 replies; 11+ messages in thread
From: KAMEZAWA Hiroyuki @ 2010-10-19  5:26 UTC (permalink / raw)
  To: Minchan Kim
  Cc: Figo.zhang, lKOSAKI Motohiro, Wu Fengguang,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

On Tue, 19 Oct 2010 14:23:29 +0900
Minchan Kim <minchan.kim@gmail.com> wrote:

> On Tue, Oct 19, 2010 at 11:59 AM, KAMEZAWA Hiroyuki
> <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> > On Tue, 19 Oct 2010 10:07:38 +0800
> > "Figo.zhang" <zhangtianfei@leadcoretech.com> wrote:
> >
> >>
> >> >
> >> > very lots of change ;)
> >> > can you please send us your crash log?
> >>
> >> i add some prink in select_bad_process() and oom_badness() to see
> >> pid/totalpages/points/memoryuseage/and finally process to selet to kill.
> >>
> >> i found it the oom-killer select: syslog-ng,mysqld,nautilus,VirtualBox
> >> to kill, so my question is:
> >>
> >> 1. the syslog-ng,mysqld,nautilus is the system foundamental process, so
> >> if oom-killer kill those process, the system will be damaged, such as
> >> lose some important data.
> >>
> >> 2. the new oom-killer just use percentage of used memory as score to
> >> select the candidate to kill, but how to know this process to very
> >> important for system?
> >>
> >
> > The kernel can never know it. Just an admin (a man or management software) knows.
> > Old kernel tries to guess it, but it tend to be wrong and many many report comes
> > "why my ....is killed..." All guesswork the kernel does is not enough, I think.
> >
> >> oom_score_adj, it is anyone commercial linux distributions to use this
> >> to protect the critical process.
> >>
> > oom_adj may be used in some system. All my customers select panic_at_oom=1
> > and cause cluster fail over rather than half-broken.
> >
> > <Off topic>
> > Your another choice is memory cgroup, I think.
> > please see documentation/cgroup/memory.txt or libcgroup.
> > http://sourceforge.net/projects/libcg/
> > You can use some fancy controls with it.
> > </Off topic>
> >
> >
> > BTW, there seems to be some strange things.
> > (CC'ed to linux-mm)
> > Brief Summary:
> > A  an oom-killer happens on swapless environment with 2.6.36-rc8.
> > A  It has 2G memory.
> > a reporter says
> > ==
> >> i want to test the oom-killer. My desktop (Dell optiplex 780, i686
> >> kernel)have 2GB ram, i turn off the swap partition, and open a huge pdf
> >> files and applications, and let the system eat huge ram.
> >>
> >> in 2.6.35, i can use ram up to 1.75GB,
> >>
> >> but in 2.6.36-rc8, i just use to 1.53GB ram , the system come very slow
> >> and crashed after some minutes , the DiskIO is very busy. i see the
> >> DiskIO read is up to 8MB/s, write just only 400KB/s, (see by conky).
> > ==
> >
> > The trigger of oom-kill is order=0 allocation. (see original mail for full log)
> >
> >
> > Oct 19 09:44:08 myhost kernel: [ A 618.441470] httpd invoked oom-killer:
> > gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
> >
> > Zone's stat is.
> >
> > Oct 19 09:44:08 myhost kernel: [ A 618.441551]
> > DMA free:7968kB min:64kB low:80kB high:96kB active_anon:3700kB inactive_anon:3752kB
> > A  A active_file:12kB inactive_file:252kB unevictable:0kB isolated(anon):0kB
> > A  A isolated(file):0kB present:15788kB mlocked:0kB dirty:0kB writeback:4kB
> > A  A mapped:52kB shmem:348kB slab_reclaimable:0kB slab_unreclaimable:16kB
> > A  A kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
> > A  A writeback_tmp:0kB pages_scanned:421 all_unreclaimable? yes
> > A  A lowmem_reserve[]: 0 865 1980 1980
> >
> > Oct 19 09:44:08 myhost kernel: [ A 618.441560]
> > Normal free:39348kB min:3728kB low:4660kB high:5592kB active_anon:176740kB
> > A  A  A  inactive_anon:25640kB active_file:84kB inactive_file:308kB
> > A  A  A  unevictable:0kB isolated(anon):0kB isolated(file):0kB present:885944kB
> > A  A  A  mlocked:0kB dirty:0kB writeback:4kB mapped:576992kB shmem:5024kB
> > A  A  A  slab_reclaimable:7612kB slab_unreclaimable:15512kB kernel_stack:2792kB
> > A  A  A  pagetables:6884kB unstable:0kB bounce:0kB writeback_tmp:0kB
> > A  A  A  pages_scanned:741 all_unreclaimable? yes
> > A  A  A  lowmem_reserve[]: 0 0 8921 8921
> >
> > Oct 19 09:44:08 myhost kernel: [ A 618.441569]
> > HighMem free:392kB min:512kB low:1712kB high:2912kB active_anon:492208kB
> > A  A  A  A inactive_anon:166404kB active_file:180kB inactive_file:840kB
> > A  A  A  A unevictable:40kB isolated(anon):0kB isolated(file):0kB present:1141984kB
> > A  A  A  A mlocked:40kB dirty:0kB writeback:12kB mapped:493648kB shmem:72216kB
> > A  A  A  A slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
> > A  A  A  A pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> > A  A  A  A pages_scanned:1552 all_unreclaimable? yes
> >
> > Highmem seems a bit strange.
> > A present(1141984) - active_anon - inactive_anon - inactive_file - active_file
> > A = 482352kB but free is 392kB.
> >
> > A Highmem is used for some other purpose than usual user's page.(pagetable is 0.)
> > A And, Hmm, mapped:493648kB seems too large for me.
> > A (active/inactive-file + shmem is not enough.)
> > A And "mapped" in NORMAL zone is large, too.
> >
> > A Does anyone have idea about file-mapped-but-not-on-LRU pages ?
> 
> Isn't it possible some file pages are much sharable?
> Please see the page_add_file_rmap.
> 

page_add_file_rmap() just counts an event where mapcount goes 0->1. 
Even if thousands process shares  a page,  it's just counted into file_mapped as 1.

Then, there are 480MB of mapped file caches. Do I miss something ?

Anyway, sum-of-all-lru-of-highmem is 480MB smaller than present pages.
and isolated(anon/file) is 0kB.
(NORMAL has similar problem)


Thanks,
-Kame




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-19  5:26           ` KAMEZAWA Hiroyuki
@ 2010-10-19  5:34             ` Minchan Kim
  2010-10-20  1:35             ` Wu Fengguang
  1 sibling, 0 replies; 11+ messages in thread
From: Minchan Kim @ 2010-10-19  5:34 UTC (permalink / raw)
  To: KAMEZAWA Hiroyuki
  Cc: Figo.zhang, lKOSAKI Motohiro, Wu Fengguang,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

On Tue, Oct 19, 2010 at 2:26 PM, KAMEZAWA Hiroyuki
<kamezawa.hiroyu@jp.fujitsu.com> wrote:
> On Tue, 19 Oct 2010 14:23:29 +0900
> Minchan Kim <minchan.kim@gmail.com> wrote:
>
>> On Tue, Oct 19, 2010 at 11:59 AM, KAMEZAWA Hiroyuki
>> <kamezawa.hiroyu@jp.fujitsu.com> wrote:
>> >
>> >  Does anyone have idea about file-mapped-but-not-on-LRU pages ?
>>
>> Isn't it possible some file pages are much sharable?
>> Please see the page_add_file_rmap.
>>

Absolutely you're right.
Today, I need sleep. :(
Sorry for the noise.


-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-19  5:26           ` KAMEZAWA Hiroyuki
  2010-10-19  5:34             ` Minchan Kim
@ 2010-10-20  1:35             ` Wu Fengguang
  2010-10-20  2:06               ` Figo.zhang
  1 sibling, 1 reply; 11+ messages in thread
From: Wu Fengguang @ 2010-10-20  1:35 UTC (permalink / raw)
  To: KAMEZAWA Hiroyuki
  Cc: Minchan Kim, Figo.zhang, lKOSAKI Motohiro,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

On Tue, Oct 19, 2010 at 01:26:40PM +0800, KAMEZAWA Hiroyuki wrote:
> On Tue, 19 Oct 2010 14:23:29 +0900
> Minchan Kim <minchan.kim@gmail.com> wrote:
> 
> > On Tue, Oct 19, 2010 at 11:59 AM, KAMEZAWA Hiroyuki
> > <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> > > On Tue, 19 Oct 2010 10:07:38 +0800
> > > "Figo.zhang" <zhangtianfei@leadcoretech.com> wrote:
> > >
> > >>
> > >> >
> > >> > very lots of change ;)
> > >> > can you please send us your crash log?
> > >>
> > >> i add some prink in select_bad_process() and oom_badness() to see
> > >> pid/totalpages/points/memoryuseage/and finally process to selet to kill.
> > >>
> > >> i found it the oom-killer select: syslog-ng,mysqld,nautilus,VirtualBox
> > >> to kill, so my question is:
> > >>
> > >> 1. the syslog-ng,mysqld,nautilus is the system foundamental process, so
> > >> if oom-killer kill those process, the system will be damaged, such as
> > >> lose some important data.
> > >>
> > >> 2. the new oom-killer just use percentage of used memory as score to
> > >> select the candidate to kill, but how to know this process to very
> > >> important for system?
> > >>
> > >
> > > The kernel can never know it. Just an admin (a man or management software) knows.
> > > Old kernel tries to guess it, but it tend to be wrong and many many report comes
> > > "why my ....is killed..." All guesswork the kernel does is not enough, I think.
> > >
> > >> oom_score_adj, it is anyone commercial linux distributions to use this
> > >> to protect the critical process.
> > >>
> > > oom_adj may be used in some system. All my customers select panic_at_oom=1
> > > and cause cluster fail over rather than half-broken.
> > >
> > > <Off topic>
> > > Your another choice is memory cgroup, I think.
> > > please see documentation/cgroup/memory.txt or libcgroup.
> > > http://sourceforge.net/projects/libcg/
> > > You can use some fancy controls with it.
> > > </Off topic>
> > >
> > >
> > > BTW, there seems to be some strange things.
> > > (CC'ed to linux-mm)
> > > Brief Summary:
> > > A  an oom-killer happens on swapless environment with 2.6.36-rc8.
> > > A  It has 2G memory.
> > > a reporter says
> > > ==
> > >> i want to test the oom-killer. My desktop (Dell optiplex 780, i686
> > >> kernel)have 2GB ram, i turn off the swap partition, and open a huge pdf
> > >> files and applications, and let the system eat huge ram.
> > >>
> > >> in 2.6.35, i can use ram up to 1.75GB,
> > >>
> > >> but in 2.6.36-rc8, i just use to 1.53GB ram , the system come very slow
> > >> and crashed after some minutes , the DiskIO is very busy. i see the
> > >> DiskIO read is up to 8MB/s, write just only 400KB/s, (see by conky).
> > > ==
> > >
> > > The trigger of oom-kill is order=0 allocation. (see original mail for full log)
> > >
> > >
> > > Oct 19 09:44:08 myhost kernel: [ A 618.441470] httpd invoked oom-killer:
> > > gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
> > >
> > > Zone's stat is.
> > >
> > > Oct 19 09:44:08 myhost kernel: [ A 618.441551]
> > > DMA free:7968kB min:64kB low:80kB high:96kB active_anon:3700kB inactive_anon:3752kB
> > > A  A active_file:12kB inactive_file:252kB unevictable:0kB isolated(anon):0kB
> > > A  A isolated(file):0kB present:15788kB mlocked:0kB dirty:0kB writeback:4kB
> > > A  A mapped:52kB shmem:348kB slab_reclaimable:0kB slab_unreclaimable:16kB
> > > A  A kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
> > > A  A writeback_tmp:0kB pages_scanned:421 all_unreclaimable? yes
> > > A  A lowmem_reserve[]: 0 865 1980 1980
> > >
> > > Oct 19 09:44:08 myhost kernel: [ A 618.441560]
> > > Normal free:39348kB min:3728kB low:4660kB high:5592kB active_anon:176740kB
> > > A  A  A  inactive_anon:25640kB active_file:84kB inactive_file:308kB
> > > A  A  A  unevictable:0kB isolated(anon):0kB isolated(file):0kB present:885944kB
> > > A  A  A  mlocked:0kB dirty:0kB writeback:4kB mapped:576992kB shmem:5024kB
> > > A  A  A  slab_reclaimable:7612kB slab_unreclaimable:15512kB kernel_stack:2792kB
> > > A  A  A  pagetables:6884kB unstable:0kB bounce:0kB writeback_tmp:0kB
> > > A  A  A  pages_scanned:741 all_unreclaimable? yes
> > > A  A  A  lowmem_reserve[]: 0 0 8921 8921
> > >
> > > Oct 19 09:44:08 myhost kernel: [ A 618.441569]
> > > HighMem free:392kB min:512kB low:1712kB high:2912kB active_anon:492208kB
> > > A  A  A  A inactive_anon:166404kB active_file:180kB inactive_file:840kB
> > > A  A  A  A unevictable:40kB isolated(anon):0kB isolated(file):0kB present:1141984kB
> > > A  A  A  A mlocked:40kB dirty:0kB writeback:12kB mapped:493648kB shmem:72216kB
> > > A  A  A  A slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
> > > A  A  A  A pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> > > A  A  A  A pages_scanned:1552 all_unreclaimable? yes
> > >
> > > Highmem seems a bit strange.
> > > A present(1141984) - active_anon - inactive_anon - inactive_file - active_file
> > > A = 482352kB but free is 392kB.
> > >
> > > A Highmem is used for some other purpose than usual user's page.(pagetable is 0.)
> > > A And, Hmm, mapped:493648kB seems too large for me.
> > > A (active/inactive-file + shmem is not enough.)
> > > A And "mapped" in NORMAL zone is large, too.
> > >
> > > A Does anyone have idea about file-mapped-but-not-on-LRU pages ?
> > 
> > Isn't it possible some file pages are much sharable?
> > Please see the page_add_file_rmap.
> > 
> 
> page_add_file_rmap() just counts an event where mapcount goes 0->1. 
> Even if thousands process shares  a page,  it's just counted into file_mapped as 1.
> 
> Then, there are 480MB of mapped file caches. Do I miss something ?
> 
> Anyway, sum-of-all-lru-of-highmem is 480MB smaller than present pages.
> and isolated(anon/file) is 0kB.
> (NORMAL has similar problem)

hugetlb files? But it's a desktop box. Figo, what's your meminfo?

The GEM objects may be files not in LRU, however they should be
accounted into shmem.

Figo, would you run "page-types -r" for some clues? It can be compiled
from the kernel tree:

        cd linux
        make Documentation/vm
        sudo Documentation/vm/page-types -r

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-20  1:35             ` Wu Fengguang
@ 2010-10-20  2:06               ` Figo.zhang
  2010-10-20  2:32                 ` KOSAKI Motohiro
  0 siblings, 1 reply; 11+ messages in thread
From: Figo.zhang @ 2010-10-20  2:06 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: KAMEZAWA Hiroyuki, Minchan Kim, lKOSAKI Motohiro,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org


> > page_add_file_rmap() just counts an event where mapcount goes 0->1. 
> > Even if thousands process shares  a page,  it's just counted into file_mapped as 1.
> > 
> > Then, there are 480MB of mapped file caches. Do I miss something ?
> > 
> > Anyway, sum-of-all-lru-of-highmem is 480MB smaller than present pages.
> > and isolated(anon/file) is 0kB.
> > (NORMAL has similar problem)
> 
> hugetlb files? But it's a desktop box. Figo, what's your meminfo?
> 
> The GEM objects may be files not in LRU, however they should be
> accounted into shmem.
> 
> Figo, would you run "page-types -r" for some clues? It can be compiled
> from the kernel tree:
> 
>         cd linux
>         make Documentation/vm
>         sudo Documentation/vm/page-types -r

hi fengguang,
here is the "page-types -r" result:

             flags	page-count       MB  symbolic-flags
long-symbolic-flags
0x0000000000000000	     16494       64
__________________________________	
0x0000000100000000	      8264       32
______________________r___________	reserved
0x0000000000010000	      3010       11
________________T_________________	compound_tail
0x0000000000008000	        76        0
_______________H__________________	compound_head
0x0000008000000000	         1        0
_____________________________c____	uncached
0x0000000400000001	         1        0
L_______________________d_________	locked,mappedtodisk
0x0000000000008014	         1        0
__R_D__________H__________________	referenced,dirty,compound_head
0x0000000000010014	        15        0
__R_D___________T_________________	referenced,dirty,compound_tail
0x0000000400000021	       235        0
L____l__________________d_________	locked,lru,mappedtodisk
0x0000000800000024	        64        0
__R__l___________________P________	referenced,lru,private
0x0000000400000028	       823        3
___U_l__________________d_________	uptodate,lru,mappedtodisk
0x0001000400000028	         2        0
___U_l__________________d_____I___	uptodate,lru,mappedtodisk,readahead
0x000000040000002c	         1        0
__RU_l__________________d_________	referenced,uptodate,lru,mappedtodisk
0x000000000000402c	      3837       14
__RU_l________b___________________	referenced,uptodate,lru,swapbacked
0x0000000800000030	         2        0
____Dl___________________P________	dirty,lru,private
0x0000000800000038	         2        0
___UDl___________________P________	uptodate,dirty,lru,private
0x0000000400000038	         2        0
___UDl__________________d_________	uptodate,dirty,lru,mappedtodisk
0x000000000000403c	        58        0
__RUDl________b___________________
referenced,uptodate,dirty,lru,swapbacked
0x0000000800000060	        53        0
_____lA__________________P________	lru,active,private
0x0000000800000064	         9        0
__R__lA__________________P________	referenced,lru,active,private
0x0000000c00000068	         8        0
___U_lA_________________dP________
uptodate,lru,active,mappedtodisk,private
0x0000000000000068	         2        0
___U_lA___________________________	uptodate,lru,active
0x000000040000006c	         1        0
__RU_lA_________________d_________
referenced,uptodate,lru,active,mappedtodisk
0x0000000800000070	         2        0
____DlA__________________P________	dirty,lru,active,private
0x0000000800000074	         9        0
__R_DlA__________________P________	referenced,dirty,lru,active,private
0x0000000000004078	     17910       69
___UDlA_______b___________________	uptodate,dirty,lru,active,swapbacked
0x000000000000407c	      5079       19
__RUDlA_______b___________________
referenced,uptodate,dirty,lru,active,swapbacked
0x000000080000007c	         1        0
__RUDlA__________________P________
referenced,uptodate,dirty,lru,active,private
0x0004000000008080	        70        0
_______S_______H________________A_	slab,compound_head,slub_frozen
0x0000000000008080	       870        3
_______S_______H__________________	slab,compound_head
0x0000000000000080	      2505        9
_______S__________________________	slab
0x0004000000000080	        51        0
_______S________________________A_	slab,slub_frozen
0x0000000800000328	         1        0
___U_l__WI_______________P________
uptodate,lru,writeback,reclaim,private
0x0000000000000400	      1724        6
__________B_______________________	buddy
0x0000000000000800	         1        0
___________M______________________	mmap
0x0000000000000804	         1        0
__R________M______________________	referenced,mmap
0x0000000400000828	       101        0
___U_l_____M____________d_________	uptodate,lru,mmap,mappedtodisk
0x000000040000082c	       150        0
__RU_l_____M____________d_________
referenced,uptodate,lru,mmap,mappedtodisk
0x0000000000004838	      4595       17
___UDl_____M__b___________________	uptodate,dirty,lru,mmap,swapbacked
0x000000000000483c	         8        0
__RUDl_____M__b___________________
referenced,uptodate,dirty,lru,mmap,swapbacked
0x0000000400000868	         3        0
___U_lA____M____________d_________	uptodate,lru,active,mmap,mappedtodisk
0x000000040000086c	       799        3
__RU_lA____M____________d_________
referenced,uptodate,lru,active,mmap,mappedtodisk
0x0000000000004878	       576        2
___UDlA____M__b___________________
uptodate,dirty,lru,active,mmap,swapbacked
0x000000000000487c	        73        0
__RUDlA____M__b___________________
referenced,uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000005808	        15        0
___U_______Ma_b___________________	uptodate,mmap,anonymous,swapbacked
0x0000000000005828	     74342      290
___U_l_____Ma_b___________________
uptodate,lru,mmap,anonymous,swapbacked
0x000000000000582c	        85        0
__RU_l_____Ma_b___________________
referenced,uptodate,lru,mmap,anonymous,swapbacked
0x000000020004582c	        12        0
__RU_l_____Ma_b___u____m__________
referenced,uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x0000000000005838	         2        0
___UDl_____Ma_b___________________
uptodate,dirty,lru,mmap,anonymous,swapbacked
0x0000000000005868	    373077     1457
___U_lA____Ma_b___________________
uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c	        48        0
__RU_lA____Ma_b___________________
referenced,uptodate,lru,active,mmap,anonymous,swapbacked
             total	    515071     2011


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-20  2:06               ` Figo.zhang
@ 2010-10-20  2:32                 ` KOSAKI Motohiro
  2010-10-20  2:58                   ` Figo.zhang
  0 siblings, 1 reply; 11+ messages in thread
From: KOSAKI Motohiro @ 2010-10-20  2:32 UTC (permalink / raw)
  To: Figo.zhang
  Cc: kosaki.motohiro, Wu Fengguang, KAMEZAWA Hiroyuki, Minchan Kim,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

> 
> > > page_add_file_rmap() just counts an event where mapcount goes 0->1. 
> > > Even if thousands process shares  a page,  it's just counted into file_mapped as 1.
> > > 
> > > Then, there are 480MB of mapped file caches. Do I miss something ?
> > > 
> > > Anyway, sum-of-all-lru-of-highmem is 480MB smaller than present pages.
> > > and isolated(anon/file) is 0kB.
> > > (NORMAL has similar problem)
> > 
> > hugetlb files? But it's a desktop box. Figo, what's your meminfo?
> > 
> > The GEM objects may be files not in LRU, however they should be
> > accounted into shmem.
> > 
> > Figo, would you run "page-types -r" for some clues? It can be compiled
> > from the kernel tree:
> > 
> >         cd linux
> >         make Documentation/vm
> >         sudo Documentation/vm/page-types -r
> 
> hi fengguang,
> here is the "page-types -r" result:
> 
>              flags	page-count       MB  symbolic-flags
> long-symbolic-flags
> 0x0000000000005828	     74342      290 ___U_l_____Ma_b___________________ uptodate,lru,mmap,anonymous,swapbacked
> 0x0000000000005868	    373077     1457 ___U_lA____Ma_b___________________ uptodate,lru,active,mmap,anonymous,swapbacked

1457+290=1747MB. that's ok. and this is very different result with your
previous oom log.

can you please try 1) invoke oom 2) get page-types -r again. I'm curious
that oom makes page accounting lost again. I mean, please send us oom 
log and "page-types -r" result.

thanks.




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-20  2:32                 ` KOSAKI Motohiro
@ 2010-10-20  2:58                   ` Figo.zhang
  2010-10-20  3:24                     ` KOSAKI Motohiro
  0 siblings, 1 reply; 11+ messages in thread
From: Figo.zhang @ 2010-10-20  2:58 UTC (permalink / raw)
  To: KOSAKI Motohiro
  Cc: Wu Fengguang, KAMEZAWA Hiroyuki, Minchan Kim,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org


> 
> can you please try 1) invoke oom 2) get page-types -r again. I'm curious
> that oom makes page accounting lost again. I mean, please send us oom 
> log and "page-types -r" result.
> 
> thanks

ok, i do the experiment and catch the log:

Oct 20 10:39:11 myhost kernel: [ 2187.700171] oom_badness: memoy use
=53, totalpages=506807, points=0
Oct 20 10:39:11 myhost kernel: [ 2187.700174] oom_badness: pid = 1280,
oom_score_adj=0, points=-30
Oct 20 10:39:11 myhost kernel: [ 2187.700176] select_bad_process,
===========have choose pid=1280 to kill, points=1
Oct 20 10:39:11 myhost kernel: [ 2187.700178] oom_badness: memoy use
=121, totalpages=506807, points=0
Oct 20 10:39:11 myhost kernel: [ 2187.700180] oom_badness: pid = 1281,
oom_score_adj=0, points=-30
Oct 20 10:39:11 myhost kernel: [ 2187.700181] oom_badness: memoy use
=229, totalpages=506807, points=0
Oct 20 10:39:11 myhost kernel: [ 2187.700183] oom_badness: pid = 1284,
oom_score_adj=0, points=0
Oct 20 10:39:11 myhost kernel: [ 2187.700185] oom_badness: memoy use
=239, totalpages=506807, points=0
Oct 20 10:39:11 myhost kernel: [ 2187.700186] oom_badness: pid = 1287,
oom_score_adj=0, points=0
Oct 20 10:39:11 myhost kernel: [ 2187.700188] oom_badness: memoy use
=71, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700190] oom_badness: pid = 1288,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700192] oom_badness: memoy use
=33, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700193] oom_badness: pid = 1317,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700195] oom_badness: memoy use
=36, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700196] oom_badness: pid = 1331,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700198] oom_badness: memoy use
=49, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700200] oom_badness: pid = 1333,
oom_score_adj=0, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700202] oom_badness: memoy use
=26, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700203] oom_badness: pid = 1396,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700205] oom_badness: memoy use
=50, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700207] oom_badness: pid = 1419,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700208] oom_badness: memoy use
=116, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700210] oom_badness: pid = 1438,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700212] oom_badness: memoy use
=21, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700213] oom_badness: pid = 1441,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700215] oom_badness: memoy use
=21, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700217] oom_badness: pid = 1442,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700219] oom_badness: memoy use
=21, totalpages=506807, points=0
Oct 20 10:39:13 myhost kernel: [ 2187.700220] oom_badness: pid = 1443,
oom_score_adj=0, points=-30
Oct 20 10:39:13 myhost kernel: [ 2187.700222] oom_badness: memoy use
=21, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700224] oom_badness: pid = 1444,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700226] oom_badness: memoy use
=20, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700227] oom_badness: pid = 1445,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700229] oom_badness: memoy use
=21, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700231] oom_badness: pid = 1446,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700233] oom_badness: memoy use
=174, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700234] oom_badness: pid = 1455,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700236] oom_badness: memoy use
=164, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700238] oom_badness: pid = 1524,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700240] oom_badness: memoy use
=251, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700241] oom_badness: pid = 1542,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700243] oom_badness: memoy use
=59, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700245] oom_badness: pid = 1562,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700246] oom_badness: memoy use
=2815, totalpages=506807, points=5
Oct 20 10:39:14 myhost kernel: [ 2187.700248] oom_badness: pid = 1656,
oom_score_adj=0, points=5
Oct 20 10:39:14 myhost kernel: [ 2187.700250] select_bad_process,
===========have choose pid=1656 to kill, points=5
Oct 20 10:39:14 myhost kernel: [ 2187.700252] oom_badness: memoy use
=56, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700253] oom_badness: pid = 1663,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700255] oom_badness: memoy use
=227, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700257] oom_badness: pid = 1666,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700258] oom_badness: memoy use
=177, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700260] oom_badness: pid = 1671,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700262] oom_badness: memoy use
=226, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700263] oom_badness: pid = 1701,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700265] oom_badness: memoy use
=1358, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700266] oom_badness: pid = 1709,
oom_score_adj=0, points=-28
Oct 20 10:39:14 myhost kernel: [ 2187.700268] oom_badness: memoy use
=1149, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700270] oom_badness: pid = 1716,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700272] oom_badness: memoy use
=28, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700273] oom_badness: pid = 1717,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700275] oom_badness: memoy use
=41, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700277] oom_badness: pid = 1723,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700278] oom_badness: memoy use
=776, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700280] oom_badness: pid = 1724,
oom_score_adj=0, points=-29
Oct 20 10:39:14 myhost kernel: [ 2187.700282] oom_badness: memoy use
=1363, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700283] oom_badness: pid = 1727,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700285] oom_badness: memoy use
=1363, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700287] oom_badness: pid = 1728,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700288] oom_badness: memoy use
=1363, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700290] oom_badness: pid = 1729,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700292] oom_badness: memoy use
=1363, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700293] oom_badness: pid = 1730,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700295] oom_badness: memoy use
=1363, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700297] oom_badness: pid = 1731,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700298] oom_badness: memoy use
=68, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700300] oom_badness: pid = 1752,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700302] oom_badness: memoy use
=192, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700303] oom_badness: pid = 1754,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700305] oom_badness: memoy use
=11195, totalpages=506807, points=22
Oct 20 10:39:14 myhost kernel: [ 2187.700306] oom_badness: pid = 1756,
oom_score_adj=0, points=-8
Oct 20 10:39:14 myhost kernel: [ 2187.700308] oom_badness: memoy use
=176, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700310] oom_badness: pid = 1775,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700312] oom_badness: memoy use
=292, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700313] oom_badness: pid = 1876,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700315] oom_badness: memoy use
=126, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700317] oom_badness: pid = 1882,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700319] oom_badness: memoy use
=110, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700321] oom_badness: pid = 1883,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700323] oom_badness: memoy use
=132, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700324] oom_badness: pid = 1950,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700326] oom_badness: memoy use
=306, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700327] oom_badness: pid = 1969,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700330] oom_badness: memoy use
=42, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700331] oom_badness: pid = 1996,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700333] oom_badness: memoy use
=378, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700334] oom_badness: pid = 1997,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700336] oom_badness: memoy use
=49, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700338] oom_badness: pid = 1999,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700340] oom_badness: memoy use
=485, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700341] oom_badness: pid = 2004,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700343] oom_badness: memoy use
=4515, totalpages=506807, points=8
Oct 20 10:39:14 myhost kernel: [ 2187.700344] oom_badness: pid = 2005,
oom_score_adj=0, points=8
Oct 20 10:39:14 myhost kernel: [ 2187.700346] select_bad_process,
===========have choose pid=2005 to kill, points=8
Oct 20 10:39:14 myhost kernel: [ 2187.700348] oom_badness: memoy use
=837, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700350] oom_badness: pid = 2014,
oom_score_adj=0, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700351] oom_badness: memoy use
=97, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700353] oom_badness: pid = 2019,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700354] oom_badness: memoy use
=833, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700356] oom_badness: pid = 2022,
oom_score_adj=0, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700358] oom_badness: memoy use
=130, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700360] oom_badness: pid = 2032,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700361] oom_badness: memoy use
=1322, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700363] oom_badness: pid = 2036,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700365] oom_badness: memoy use
=172, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700366] oom_badness: pid = 2039,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700368] oom_badness: memoy use
=166, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700370] oom_badness: pid = 2041,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700371] oom_badness: memoy use
=51, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700373] oom_badness: pid = 2042,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700375] oom_badness: memoy use
=5378, totalpages=506807, points=10
Oct 20 10:39:14 myhost kernel: [ 2187.700376] oom_badness: pid = 2046,
oom_score_adj=0, points=10
Oct 20 10:39:14 myhost kernel: [ 2187.700378] select_bad_process,
===========have choose pid=2046 to kill, points=10
Oct 20 10:39:14 myhost kernel: [ 2187.700380] oom_badness: memoy use
=140, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700381] oom_badness: pid = 2048,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700383] oom_badness: memoy use
=1188, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700385] oom_badness: pid = 2056,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700386] oom_badness: memoy use
=584, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700388] oom_badness: pid = 2059,
oom_score_adj=0, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700390] oom_badness: memoy use
=336, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700391] oom_badness: pid = 2062,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700393] oom_badness: memoy use
=924, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700395] oom_badness: pid = 2063,
oom_score_adj=0, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700396] oom_badness: memoy use
=369, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700398] oom_badness: pid = 2065,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700400] oom_badness: memoy use
=791, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700401] oom_badness: pid = 2066,
oom_score_adj=0, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700403] oom_badness: memoy use
=70, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700405] oom_badness: pid = 2067,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700406] oom_badness: memoy use
=41485, totalpages=506807, points=81
Oct 20 10:39:14 myhost kernel: [ 2187.700408] oom_badness: pid = 2069,
oom_score_adj=0, points=81
Oct 20 10:39:14 myhost kernel: [ 2187.700409] select_bad_process,
===========have choose pid=2069 to kill, points=81
Oct 20 10:39:14 myhost kernel: [ 2187.700411] oom_badness: memoy use
=198, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700413] oom_badness: pid = 2077,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700415] oom_badness: memoy use
=1599, totalpages=506807, points=3
Oct 20 10:39:14 myhost kernel: [ 2187.700416] oom_badness: pid = 2079,
oom_score_adj=0, points=3
Oct 20 10:39:14 myhost kernel: [ 2187.700418] oom_badness: memoy use
=416, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700420] oom_badness: pid = 2081,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700422] oom_badness: memoy use
=264, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700423] oom_badness: pid = 2086,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700425] oom_badness: memoy use
=287, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700426] oom_badness: pid = 2087,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700428] oom_badness: memoy use
=291, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700430] oom_badness: pid = 2088,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700432] oom_badness: memoy use
=434, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700433] oom_badness: pid = 2089,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700435] oom_badness: memoy use
=269, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700436] oom_badness: pid = 2099,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700438] oom_badness: memoy use
=376, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700440] oom_badness: pid = 2101,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700441] oom_badness: memoy use
=995, totalpages=506807, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700443] oom_badness: pid = 2110,
oom_score_adj=0, points=1
Oct 20 10:39:14 myhost kernel: [ 2187.700445] oom_badness: memoy use
=276, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700446] oom_badness: pid = 2113,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700448] oom_badness: memoy use
=177, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700450] oom_badness: pid = 2115,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700451] oom_badness: memoy use
=69, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700453] oom_badness: pid = 2118,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700455] oom_badness: memoy use
=119, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700456] oom_badness: pid = 2129,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700458] oom_badness: memoy use
=2216, totalpages=506807, points=4
Oct 20 10:39:14 myhost kernel: [ 2187.700460] oom_badness: pid = 2133,
oom_score_adj=0, points=-26
Oct 20 10:39:14 myhost kernel: [ 2187.700462] oom_badness: memoy use
=202, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700463] oom_badness: pid = 2145,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700465] oom_badness: memoy use
=213, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700466] oom_badness: pid = 2157,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700468] oom_badness: memoy use
=1183, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700470] oom_badness: pid = 2189,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700471] oom_badness: memoy use
=29, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700473] oom_badness: pid = 2191,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700475] oom_badness: memoy use
=451, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700476] oom_badness: pid = 2193,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700478] oom_badness: memoy use
=36, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700479] oom_badness: pid = 2205,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700481] oom_badness: memoy use
=476, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700483] oom_badness: pid = 2206,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700484] oom_badness: memoy use
=43, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700486] oom_badness: pid = 2277,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700488] oom_badness: memoy use
=3431, totalpages=506807, points=6
Oct 20 10:39:14 myhost kernel: [ 2187.700489] oom_badness: pid = 2278,
oom_score_adj=0, points=-24
Oct 20 10:39:14 myhost kernel: [ 2187.700491] oom_badness: memoy use
=43, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700493] oom_badness: pid = 2282,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700495] oom_badness: memoy use
=51, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700496] oom_badness: pid = 2283,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700498] oom_badness: memoy use
=412, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700500] oom_badness: pid = 2286,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700502] oom_badness: memoy use
=1891, totalpages=506807, points=3
Oct 20 10:39:14 myhost kernel: [ 2187.700503] oom_badness: pid = 2319,
oom_score_adj=0, points=3
Oct 20 10:39:14 myhost kernel: [ 2187.700505] oom_badness: memoy use
=21095, totalpages=506807, points=41
Oct 20 10:39:14 myhost kernel: [ 2187.700506] oom_badness: pid = 2403,
oom_score_adj=0, points=41
Oct 20 10:39:14 myhost kernel: [ 2187.700508] oom_badness: memoy use
=2021, totalpages=506807, points=3
Oct 20 10:39:14 myhost kernel: [ 2187.700510] oom_badness: pid = 2478,
oom_score_adj=0, points=3
Oct 20 10:39:14 myhost kernel: [ 2187.700511] oom_badness: memoy use
=1263, totalpages=506807, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700513] oom_badness: pid = 2487,
oom_score_adj=0, points=2
Oct 20 10:39:14 myhost kernel: [ 2187.700515] oom_badness: memoy use
=191, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700516] oom_badness: pid = 2614,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700518] oom_badness: memoy use
=7232, totalpages=506807, points=14
Oct 20 10:39:14 myhost kernel: [ 2187.700519] oom_badness: pid = 2658,
oom_score_adj=0, points=14
Oct 20 10:39:14 myhost kernel: [ 2187.700521] oom_badness: memoy use
=327, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700523] oom_badness: pid = 2784,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700525] oom_badness: memoy use
=479, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700526] oom_badness: pid = 2796,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700528] oom_badness: memoy use
=3098, totalpages=506807, points=6
Oct 20 10:39:14 myhost kernel: [ 2187.700529] oom_badness: pid = 2882,
oom_score_adj=0, points=6
Oct 20 10:39:14 myhost kernel: [ 2187.700532] oom_badness: memoy use
=36, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700533] oom_badness: pid = 3319,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700535] oom_badness: memoy use
=478, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700536] oom_badness: pid = 3320,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700538] oom_badness: memoy use
=15539, totalpages=506807, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700540] oom_badness: pid = 3436,
oom_score_adj=0, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700541] oom_badness: memoy use
=118, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700543] oom_badness: pid = 3440,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700544] oom_badness: memoy use
=29031, totalpages=506807, points=57
Oct 20 10:39:14 myhost kernel: [ 2187.700546] oom_badness: pid = 3446,
oom_score_adj=0, points=57
Oct 20 10:39:14 myhost kernel: [ 2187.700548] oom_badness: memoy use
=15735, totalpages=506807, points=31
Oct 20 10:39:14 myhost kernel: [ 2187.700549] oom_badness: pid = 3498,
oom_score_adj=0, points=31
Oct 20 10:39:14 myhost kernel: [ 2187.700551] oom_badness: memoy use
=66, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700552] oom_badness: pid = 3506,
oom_score_adj=0, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700554] oom_badness: memoy use
=4007, totalpages=506807, points=7
Oct 20 10:39:14 myhost kernel: [ 2187.700556] oom_badness: pid = 3508,
oom_score_adj=0, points=7
Oct 20 10:39:14 myhost kernel: [ 2187.700557] oom_badness: memoy use
=17404, totalpages=506807, points=34
Oct 20 10:39:14 myhost kernel: [ 2187.700559] oom_badness: pid = 3564,
oom_score_adj=0, points=34
Oct 20 10:39:14 myhost kernel: [ 2187.700560] oom_badness: memoy use
=14779, totalpages=506807, points=29
Oct 20 10:39:14 myhost kernel: [ 2187.700562] oom_badness: pid = 3622,
oom_score_adj=0, points=29
Oct 20 10:39:14 myhost kernel: [ 2187.700563] oom_badness: memoy use
=14386, totalpages=506807, points=28
Oct 20 10:39:14 myhost kernel: [ 2187.700565] oom_badness: pid = 3692,
oom_score_adj=0, points=28
Oct 20 10:39:14 myhost kernel: [ 2187.700567] oom_badness: memoy use
=13580, totalpages=506807, points=26
Oct 20 10:39:14 myhost kernel: [ 2187.700568] oom_badness: pid = 3708,
oom_score_adj=0, points=26
Oct 20 10:39:14 myhost kernel: [ 2187.700570] oom_badness: memoy use
=16011, totalpages=506807, points=31
Oct 20 10:39:14 myhost kernel: [ 2187.700571] oom_badness: pid = 3770,
oom_score_adj=0, points=31
Oct 20 10:39:14 myhost kernel: [ 2187.700573] oom_badness: memoy use
=19788, totalpages=506807, points=39
Oct 20 10:39:14 myhost kernel: [ 2187.700574] oom_badness: pid = 3889,
oom_score_adj=0, points=39
Oct 20 10:39:14 myhost kernel: [ 2187.700576] oom_badness: memoy use
=23319, totalpages=506807, points=46
Oct 20 10:39:14 myhost kernel: [ 2187.700577] oom_badness: pid = 3903,
oom_score_adj=0, points=46
Oct 20 10:39:14 myhost kernel: [ 2187.700579] oom_badness: memoy use
=14994, totalpages=506807, points=29
Oct 20 10:39:14 myhost kernel: [ 2187.700580] oom_badness: pid = 3927,
oom_score_adj=0, points=29
Oct 20 10:39:14 myhost kernel: [ 2187.700582] oom_badness: memoy use
=15434, totalpages=506807, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700583] oom_badness: pid = 3940,
oom_score_adj=0, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700585] oom_badness: memoy use
=17479, totalpages=506807, points=34
Oct 20 10:39:14 myhost kernel: [ 2187.700586] oom_badness: pid = 3954,
oom_score_adj=0, points=34
Oct 20 10:39:14 myhost kernel: [ 2187.700588] oom_badness: memoy use
=16979, totalpages=506807, points=33
Oct 20 10:39:14 myhost kernel: [ 2187.700589] oom_badness: pid = 3972,
oom_score_adj=0, points=33
Oct 20 10:39:14 myhost kernel: [ 2187.700591] oom_badness: memoy use
=13305, totalpages=506807, points=26
Oct 20 10:39:14 myhost kernel: [ 2187.700593] oom_badness: pid = 4002,
oom_score_adj=0, points=26
Oct 20 10:39:14 myhost kernel: [ 2187.700594] oom_badness: memoy use
=16434, totalpages=506807, points=32
Oct 20 10:39:14 myhost kernel: [ 2187.700596] oom_badness: pid = 4015,
oom_score_adj=0, points=32
Oct 20 10:39:14 myhost kernel: [ 2187.700597] oom_badness: memoy use
=15376, totalpages=506807, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700599] oom_badness: pid = 4028,
oom_score_adj=0, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700601] oom_badness: memoy use
=13587, totalpages=506807, points=26
Oct 20 10:39:14 myhost kernel: [ 2187.700602] oom_badness: pid = 4081,
oom_score_adj=0, points=26
Oct 20 10:39:14 myhost kernel: [ 2187.700604] oom_badness: memoy use
=36, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700605] oom_badness: pid = 4085,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700607] oom_badness: memoy use
=36, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700609] oom_badness: pid = 4086,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700611] oom_badness: memoy use
=35, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700612] oom_badness: pid = 4089,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700614] oom_badness: memoy use
=43, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700615] oom_badness: pid = 4092,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700617] oom_badness: memoy use
=42, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700619] oom_badness: pid = 4093,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700620] oom_badness: memoy use
=55, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700622] oom_badness: pid = 4096,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700624] oom_badness: memoy use
=54, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700625] oom_badness: pid = 4097,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700627] oom_badness: memoy use
=58, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700629] oom_badness: pid = 4128,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700630] oom_badness: memoy use
=58, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700632] oom_badness: pid = 4129,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700634] oom_badness: memoy use
=86, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700636] oom_badness: pid = 4130,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700637] oom_badness: memoy use
=87, totalpages=506807, points=0
Oct 20 10:39:14 myhost kernel: [ 2187.700639] oom_badness: pid = 4131,
oom_score_adj=0, points=-30
Oct 20 10:39:14 myhost kernel: [ 2187.700641] oom_badness: memoy use
=15280, totalpages=506807, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700642] oom_badness: pid = 4141,
oom_score_adj=0, points=30
Oct 20 10:39:14 myhost kernel: [ 2187.700644] acroread invoked
oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Oct 20 10:39:14 myhost kernel: [ 2187.700647] acroread cpuset=/
mems_allowed=0
Oct 20 10:39:14 myhost kernel: [ 2187.700649] Pid: 3446, comm: acroread
Not tainted 2.6.36testing #7
Oct 20 10:39:14 myhost kernel: [ 2187.700650] Call Trace:
Oct 20 10:39:14 myhost kernel: [ 2187.700656]  [<c10c0a20>]
dump_header.clone.5+0x80/0x1e0
Oct 20 10:39:14 myhost kernel: [ 2187.700660]  [<c12eb5b6>] ? printk
+0x18/0x1a
Oct 20 10:39:14 myhost kernel: [ 2187.700662]  [<c10c0d85>] ?
oom_badness+0x185/0x1a0
Oct 20 10:39:14 myhost kernel: [ 2187.700664]  [<c10c0dfc>]
oom_kill_process+0x5c/0x210
Oct 20 10:39:14 myhost kernel: [ 2187.700666]  [<c10c1031>] ?
select_bad_process.clone.7+0x81/0x100
Oct 20 10:39:14 myhost kernel: [ 2187.700668]  [<c10c134f>]
out_of_memory+0xbf/0x1d0
Oct 20 10:39:14 myhost kernel: [ 2187.700670]  [<c10c1218>] ?
try_set_zonelist_oom+0xc8/0xe0
Oct 20 10:39:14 myhost kernel: [ 2187.700673]  [<c10c4bd8>]
__alloc_pages_nodemask+0x5e8/0x600
Oct 20 10:39:14 myhost kernel: [ 2187.700676]  [<c10c6585>]
__do_page_cache_readahead+0x105/0x230
Oct 20 10:39:14 myhost kernel: [ 2187.700678]  [<c10c6911>] ra_submit
+0x21/0x30
Oct 20 10:39:14 myhost kernel: [ 2187.700680]  [<c10beb8b>]
filemap_fault+0x36b/0x3e0
Oct 20 10:39:14 myhost kernel: [ 2187.700683]  [<c10d5cab>] __do_fault
+0x3b/0x4f0
Oct 20 10:39:14 myhost kernel: [ 2187.700686]  [<c10d8d0d>]
handle_mm_fault+0xfd/0x930
Oct 20 10:39:14 myhost kernel: [ 2187.700689]  [<c1029250>] ?
do_page_fault+0x0/0x3e0
Oct 20 10:39:14 myhost kernel: [ 2187.700691]  [<c10293a0>]
do_page_fault+0x150/0x3e0
Oct 20 10:39:14 myhost kernel: [ 2187.700694]  [<c106a69f>] ?
ktime_get_ts+0xff/0x130
Oct 20 10:39:14 myhost kernel: [ 2187.700697]  [<c1109ce4>] ? sys_poll
+0x54/0xd0
Oct 20 10:39:14 myhost kernel: [ 2187.700699]  [<c1029250>] ?
do_page_fault+0x0/0x3e0
Oct 20 10:39:14 myhost kernel: [ 2187.700702]  [<c12ef1fb>] error_code
+0x67/0x6c
Oct 20 10:39:14 myhost kernel: [ 2187.700703] Mem-Info:
Oct 20 10:39:14 myhost kernel: [ 2187.700704] DMA per-cpu:
Oct 20 10:39:14 myhost kernel: [ 2187.700706] CPU    0: hi:    0, btch:
1 usd:   0
Oct 20 10:39:14 myhost kernel: [ 2187.700707] CPU    1: hi:    0, btch:
1 usd:   0
Oct 20 10:39:14 myhost kernel: [ 2187.700708] Normal per-cpu:
Oct 20 10:39:14 myhost kernel: [ 2187.700709] CPU    0: hi:  186, btch:
31 usd:   0
Oct 20 10:39:14 myhost kernel: [ 2187.700711] CPU    1: hi:  186, btch:
31 usd:  40
Oct 20 10:39:14 myhost kernel: [ 2187.700712] HighMem per-cpu:
Oct 20 10:39:14 myhost kernel: [ 2187.700713] CPU    0: hi:  186, btch:
31 usd:   0
Oct 20 10:39:14 myhost kernel: [ 2187.700714] CPU    1: hi:  186, btch:
31 usd:  52
Oct 20 10:39:14 myhost kernel: [ 2187.700718] active_anon:398375
inactive_anon:82967 isolated_anon:0
Oct 20 10:39:14 myhost kernel: [ 2187.700718]  active_file:81
inactive_file:429 isolated_file:32
Oct 20 10:39:14 myhost kernel: [ 2187.700719]  unevictable:13 dirty:2
writeback:14 unstable:0
Oct 20 10:39:14 myhost kernel: [ 2187.700720]  free:11942
slab_reclaimable:2391 slab_unreclaimable:3303
Oct 20 10:39:14 myhost kernel: [ 2187.700721]  mapped:5617 shmem:33909
pagetables:2280 bounce:0
Oct 20 10:39:14 myhost kernel: [ 2187.700725] DMA free:7984kB min:64kB
low:80kB high:96kB active_anon:3852kB inactive_anon:3968kB
active_file:0kB inactive_file:52kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15788kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:36 all_unreclaimable? no
Oct 20 10:39:14 myhost kernel: [ 2187.700728] lowmem_reserve[]: 0 865
1980 1980
Oct 20 10:39:14 myhost kernel: [ 2187.700734] Normal free:39404kB
min:3728kB low:4660kB high:5592kB active_anon:732980kB
inactive_anon:42036kB active_file:116kB inactive_file:780kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:885944kB
mlocked:0kB dirty:8kB writeback:48kB mapped:6728kB shmem:44600kB
slab_reclaimable:9564kB slab_unreclaimable:13196kB kernel_stack:3200kB
pagetables:9120kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:1207 all_unreclaimable? no
Oct 20 10:39:14 myhost kernel: [ 2187.700737] lowmem_reserve[]: 0 0 8921
8921
Oct 20 10:39:14 myhost kernel: [ 2187.700743] HighMem free:380kB
min:512kB low:1712kB high:2912kB active_anon:856668kB
inactive_anon:285864kB active_file:208kB inactive_file:896kB
unevictable:52kB isolated(anon):0kB isolated(file):0kB present:1141984kB
mlocked:52kB dirty:0kB writeback:8kB mapped:15740kB shmem:91036kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:1398 all_unreclaimable? no
Oct 20 10:39:14 myhost kernel: [ 2187.700746] lowmem_reserve[]: 0 0 0 0
Oct 20 10:39:14 myhost kernel: [ 2187.700749] DMA: 1*4kB 1*8kB 1*16kB
3*32kB 1*64kB 3*128kB 3*256kB 3*512kB 1*1024kB 0*2048kB 1*4096kB =
7996kB
Oct 20 10:39:14 myhost kernel: [ 2187.700755] Normal: 53*4kB 79*8kB
144*16kB 139*32kB 121*64kB 44*128kB 40*256kB 6*512kB 1*1024kB 0*2048kB
1*4096kB = 39404kB
Oct 20 10:39:14 myhost kernel: [ 2187.700761] HighMem: 29*4kB 17*8kB
6*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB
= 380kB
Oct 20 10:39:14 myhost kernel: [ 2187.700767] 34452 total pagecache
pages
Oct 20 10:39:14 myhost kernel: [ 2187.700768] 0 pages in swap cache
Oct 20 10:39:14 myhost kernel: [ 2187.700769] Swap cache stats: add 0,
delete 0, find 0/0
Oct 20 10:39:14 myhost kernel: [ 2187.700771] Free swap  = 0kB
Oct 20 10:39:14 myhost kernel: [ 2187.700771] Total swap = 0kB
Oct 20 10:39:14 myhost kernel: [ 2187.704392] 515070 pages RAM
Oct 20 10:39:14 myhost kernel: [ 2187.704393] 287745 pages HighMem
Oct 20 10:39:14 myhost kernel: [ 2187.704394] 8264 pages reserved
Oct 20 10:39:14 myhost kernel: [ 2187.704395] 29765 pages shared
Oct 20 10:39:14 myhost kernel: [ 2187.704396] 478272 pages non-shared
Oct 20 10:39:14 myhost kernel: [ 2187.704397] [ pid ]   uid  tgid
total_vm      rss cpu oom_adj oom_score_adj name
Oct 20 10:39:14 myhost kernel: [ 2187.704404] [  566]     0   566
578      167   0     -17         -1000 udevd
Oct 20 10:39:14 myhost kernel: [ 2187.704407] [ 1280]     0  1280
1280       53   1       0             0 syslog-ng
Oct 20 10:39:14 myhost kernel: [ 2187.704410] [ 1281]     0  1281
1358      121   1       0             0 syslog-ng
Oct 20 10:39:14 myhost kernel: [ 2187.704412] [ 1284]    81  1284
780      229   0       0             0 dbus-daemon
Oct 20 10:39:14 myhost kernel: [ 2187.704415] [ 1287]    82  1287
3757      239   0       0             0 hald
Oct 20 10:39:14 myhost kernel: [ 2187.704418] [ 1288]     0  1288
898       71   0       0             0 hald-runner
Oct 20 10:39:14 myhost kernel: [ 2187.704420] [ 1317]     0  1317
914       33   1       0             0 hald-addon-inpu
Oct 20 10:39:14 myhost kernel: [ 2187.704423] [ 1331]     0  1331
914       36   1       0             0 hald-addon-stor
Oct 20 10:39:14 myhost kernel: [ 2187.704425] [ 1333]    82  1333
823       49   0       0             0 hald-addon-acpi
Oct 20 10:39:14 myhost kernel: [ 2187.704428] [ 1370]     0  1370
577      168   1     -17         -1000 udevd
Oct 20 10:39:14 myhost kernel: [ 2187.704430] [ 1396]     0  1396
451       26   1       0             0 crond
Oct 20 10:39:14 myhost kernel: [ 2187.704433] [ 1419]     0  1419
718       50   1       0             0 mysqld_safe
Oct 20 10:39:14 myhost kernel: [ 2187.704436] [ 1438]     0  1438
3575      116   0       0             0 gdm-binary
Oct 20 10:39:14 myhost kernel: [ 2187.704438] [ 1441]     0  1441
439       21   1       0             0 agetty
Oct 20 10:39:14 myhost kernel: [ 2187.704441] [ 1442]     0  1442
439       21   0       0             0 agetty
Oct 20 10:39:14 myhost kernel: [ 2187.704443] [ 1443]     0  1443
439       21   1       0             0 agetty
Oct 20 10:39:14 myhost kernel: [ 2187.704446] [ 1444]     0  1444
439       21   0       0             0 agetty
Oct 20 10:39:14 myhost kernel: [ 2187.704448] [ 1445]     0  1445
439       20   0       0             0 agetty
Oct 20 10:39:14 myhost kernel: [ 2187.704451] [ 1446]     0  1446
439       21   1       0             0 agetty
Oct 20 10:39:14 myhost kernel: [ 2187.704453] [ 1455]     0  1455
2214      174   0       0             0 cupsd
Oct 20 10:39:14 myhost kernel: [ 2187.704456] [ 1517]     0  1517
1647       98   1     -17         -1000 sshd
Oct 20 10:39:14 myhost kernel: [ 2187.704459] [ 1524]     0  1524
6451      164   1       0             0 NetworkManager
Oct 20 10:39:14 myhost kernel: [ 2187.704461] [ 1542]     0  1542
5710      251   0       0             0 polkitd
Oct 20 10:39:14 myhost kernel: [ 2187.704464] [ 1562]     0  1562
2014       59   1       0             0 vmware-usbarbit
Oct 20 10:39:14 myhost kernel: [ 2187.704466] [ 1656]    89  1656
29902     2815   0       0             0 mysqld
Oct 20 10:39:14 myhost kernel: [ 2187.704469] [ 1663]     0  1663
1247       56   0       0             0 wpa_supplicant
Oct 20 10:39:14 myhost kernel: [ 2187.704471] [ 1666]     0  1666
4910      227   0       0             0 smbd
Oct 20 10:39:14 myhost kernel: [ 2187.704474] [ 1671]     0  1671
2807      177   0       0             0 nmbd
Oct 20 10:39:14 myhost kernel: [ 2187.704476] [ 1701]     0  1701
4910      226   1       0             0 smbd
Oct 20 10:39:14 myhost kernel: [ 2187.704479] [ 1709]     0  1709
5162     1358   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704482] [ 1716]    33  1716
4841     1149   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704484] [ 1717]     0  1717
487       28   0       0             0 dhcpcd
Oct 20 10:39:14 myhost kernel: [ 2187.704487] [ 1723]     0  1723
946       41   0       0             0 ApplicationPool
Oct 20 10:39:14 myhost kernel: [ 2187.704489] [ 1724]     0  1724
3555      776   1       0             0 ruby
Oct 20 10:39:14 myhost kernel: [ 2187.704492] [ 1727]    33  1727
5162     1363   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704494] [ 1728]    33  1728
5162     1363   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704496] [ 1729]    33  1729
5162     1363   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704499] [ 1730]    33  1730
5162     1363   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704501] [ 1731]    33  1731
5162     1363   0       0             0 httpd
Oct 20 10:39:14 myhost kernel: [ 2187.704504] [ 1752]     0  1752
854       68   0       0             0 cntlm
Oct 20 10:39:14 myhost kernel: [ 2187.704506] [ 1754]     0  1754
4391      192   0       0             0 gdm-simple-slav
Oct 20 10:39:14 myhost kernel: [ 2187.704509] [ 1756]     0  1756
38002    11195   0       0             0 Xorg
Oct 20 10:39:14 myhost kernel: [ 2187.704512] [ 1775]     0  1775
6671      176   0       0             0 console-kit-dae
Oct 20 10:39:14 myhost kernel: [ 2187.704514] [ 1876]   120  1876
6572      292   0       0             0 polkit-gnome-au
Oct 20 10:39:14 myhost kernel: [ 2187.704517] [ 1882]     0  1882
3666      126   0       0             0 upowerd
Oct 20 10:39:14 myhost kernel: [ 2187.704520] [ 1883]     0  1883
3900      110   1       0             0 gdm-session-wor
Oct 20 10:39:14 myhost kernel: [ 2187.704522] [ 1950]  1000  1950
7737      132   1       0             0 gnome-keyring-d
Oct 20 10:39:14 myhost kernel: [ 2187.704525] [ 1969]  1000  1969
9233      306   1       0             0 gnome-session
Oct 20 10:39:14 myhost kernel: [ 2187.704528] [ 1996]  1000  1996
794       42   0       0             0 dbus-launch
Oct 20 10:39:14 myhost kernel: [ 2187.704530] [ 1997]  1000  1997
1249      378   0       0             0 dbus-daemon
Oct 20 10:39:14 myhost kernel: [ 2187.704533] [ 1999]  1000  1999
886       49   1       0             0 ssh-agent
Oct 20 10:39:14 myhost kernel: [ 2187.704535] [ 2004]  1000  2004
2650      485   1       0             0 gconfd-2
Oct 20 10:39:14 myhost kernel: [ 2187.704538] [ 2005]  1000  2005
11022     4515   0       0             0 fcitx
Oct 20 10:39:14 myhost kernel: [ 2187.704541] [ 2014]  1000  2014
8792      837   0       0             0 gnome-settings-
Oct 20 10:39:14 myhost kernel: [ 2187.704543] [ 2019]  1000  2019
2201       97   1       0             0 gvfsd
Oct 20 10:39:14 myhost kernel: [ 2187.704546] [ 2022]  1000  2022
40893      833   1       0             0 metacity
Oct 20 10:39:14 myhost kernel: [ 2187.704549] [ 2028]     0  2028
577      161   0     -17         -1000 udevd
Oct 20 10:39:14 myhost kernel: [ 2187.704551] [ 2032]  1000  2032
7599      130   0       0             0 gvfs-fuse-daemo
Oct 20 10:39:14 myhost kernel: [ 2187.704554] [ 2036]  1000  2036
46396     1322   0       0             0 gnome-panel
Oct 20 10:39:14 myhost kernel: [ 2187.704556] [ 2039]  1000  2039
8657      172   0       0             0 gvfs-gdu-volume
Oct 20 10:39:14 myhost kernel: [ 2187.704559] [ 2041]     0  2041
5721      166   0       0             0 udisks-daemon
Oct 20 10:39:14 myhost kernel: [ 2187.704561] [ 2042]     0  2042
1284       51   0       0             0 udisks-daemon
Oct 20 10:39:14 myhost kernel: [ 2187.704564] [ 2046]  1000  2046
63112     5378   0       0             0 nautilus
Oct 20 10:39:14 myhost kernel: [ 2187.704567] [ 2048]  1000  2048
9063      140   0       0             0 bonobo-activati
Oct 20 10:39:14 myhost kernel: [ 2187.704569] [ 2056]  1000  2056
43835     1188   1       0             0 wnck-applet
Oct 20 10:39:14 myhost kernel: [ 2187.704572] [ 2059]  1000  2059
42284      583   1       0             0 cpufreq-applet
Oct 20 10:39:14 myhost kernel: [ 2187.704574] [ 2062]  1000  2062
41008      336   1       0             0 notification-ar
Oct 20 10:39:14 myhost kernel: [ 2187.704577] [ 2063]  1000  2063
43602      924   0       0             0 mixer_applet2
Oct 20 10:39:14 myhost kernel: [ 2187.704580] [ 2065]  1000  2065
41469      369   0       0             0 multiload-apple
Oct 20 10:39:14 myhost kernel: [ 2187.704582] [ 2066]  1000  2066
45661      791   1       0             0 clock-applet
Oct 20 10:39:14 myhost kernel: [ 2187.704585] [ 2067]  1000  2067
1631       70   0       0             0 sh
Oct 20 10:39:14 myhost kernel: [ 2187.704587] [ 2069]  1000  2069
114532    41485   0       0             0 evolution
Oct 20 10:39:14 myhost kernel: [ 2187.704590] [ 2077]  1000  2077
6808      198   0       0             0 polkit-gnome-au
Oct 20 10:39:14 myhost kernel: [ 2187.704592] [ 2079]  1000  2079
8439     1599   0       0             0 applet.py
Oct 20 10:39:14 myhost kernel: [ 2187.704595] [ 2081]  1000  2081
10737      416   1       0             0 evolution-alarm
Oct 20 10:39:14 myhost kernel: [ 2187.704598] [ 2086]  1000  2086
5037      264   0       0             0 gdu-notificatio
Oct 20 10:39:14 myhost kernel: [ 2187.704600] [ 2087]  1000  2087
7018      287   1       0             0 gnome-power-man
Oct 20 10:39:14 myhost kernel: [ 2187.704603] [ 2088]  1000  2088
7574      291   0       0             0 vino-server
Oct 20 10:39:14 myhost kernel: [ 2187.704605] [ 2089]  1000  2089
72305      434   0       0             0 nm-applet
Oct 20 10:39:14 myhost kernel: [ 2187.704608] [ 2099]  1000  2099
7247      269   1       0             0 gnome-screensav
Oct 20 10:39:14 myhost kernel: [ 2187.704611] [ 2101]  1000  2101
15260      376   1       0             0 e-calendar-fact
Oct 20 10:39:14 myhost kernel: [ 2187.704613] [ 2110]  1000  2110
41625      995   1       0             0 notify-osd
Oct 20 10:39:14 myhost kernel: [ 2187.704616] [ 2113]  1000  2113
2414      276   0       0             0 gvfsd-trash
Oct 20 10:39:14 myhost kernel: [ 2187.704618] [ 2115]  1000  2115
1806      177   1       0             0 mission-control
Oct 20 10:39:14 myhost kernel: [ 2187.704621] [ 2118]     0  2118
3290       69   0       0             0 system-tools-ba
Oct 20 10:39:14 myhost kernel: [ 2187.704624] [ 2129]  1000  2129
2201      119   0       0             0 gvfsd-burn
Oct 20 10:39:14 myhost kernel: [ 2187.704626] [ 2133]     0  2133
3296     2216   1       0             0 SystemToolsBack
Oct 20 10:39:14 myhost kernel: [ 2187.704629] [ 2145]  1000  2145
2299      202   1       0             0 gvfsd-metadata
Oct 20 10:39:14 myhost kernel: [ 2187.704631] [ 2157]  1000  2157
22558      213   1       0             0 conky
Oct 20 10:39:14 myhost kernel: [ 2187.704634] [ 2189]  1000  2189
43833     1183   1       0             0 gnome-terminal
Oct 20 10:39:14 myhost kernel: [ 2187.704636] [ 2191]  1000  2191
450       29   1       0             0 gnome-pty-helpe
Oct 20 10:39:14 myhost kernel: [ 2187.704639] [ 2193]  1000  2193
2041      451   1       0             0 bash
Oct 20 10:39:14 myhost kernel: [ 2187.704641] [ 2205]  1000  2205
1480       36   1       0             0 su
Oct 20 10:39:14 myhost kernel: [ 2187.704644] [ 2206]     0  2206
2041      476   1       0             0 bash
Oct 20 10:39:14 myhost kernel: [ 2187.704646] [ 2277]     0  2277
1487       43   1       0             0 sudo
Oct 20 10:39:14 myhost kernel: [ 2187.704649] [ 2278]     0  2278
55057     3431   0       0             0 gedit
Oct 20 10:39:14 myhost kernel: [ 2187.704651] [ 2282]     0  2282
794       43   1       0             0 dbus-launch
Oct 20 10:39:14 myhost kernel: [ 2187.704654] [ 2283]     0  2283
593       51   0       0             0 dbus-daemon
Oct 20 10:39:14 myhost kernel: [ 2187.704657] [ 2286]     0  2286
2590      412   1       0             0 gconfd-2
Oct 20 10:39:14 myhost kernel: [ 2187.704659] [ 2319]  1000  2319
151661     1891   1       0             0 stardict
Oct 20 10:39:14 myhost kernel: [ 2187.704662] [ 2403]  1000  2403
104374    21095   0       0             0 firefox
Oct 20 10:39:14 myhost kernel: [ 2187.704664] [ 2478]  1000  2478
23809     2021   1       0             0 plugin-containe
Oct 20 10:39:14 myhost kernel: [ 2187.704667] [ 2487]  1000  2487
9147     1263   0       0             0 GoogleTalkPlugi
Oct 20 10:39:14 myhost kernel: [ 2187.704669] [ 2614]  1000  2614
5541      191   0       0             0 dconf-service
Oct 20 10:39:14 myhost kernel: [ 2187.704672] [ 2658]  1000  2658
57278     7232   1       0             0 skype
Oct 20 10:39:14 myhost kernel: [ 2187.704674] [ 2784]  1000  2784
9883      327   0       0             0 gvfsd-smb
Oct 20 10:39:14 myhost kernel: [ 2187.704677] [ 2796]  1000  2796
2067      479   0       0             0 bash
Oct 20 10:39:14 myhost kernel: [ 2187.704680] [ 2882]  1000  2882
50484     3098   1       0             0 gedit
Oct 20 10:39:14 myhost kernel: [ 2187.704682] [ 3319]  1000  3319
1480       36   1       0             0 su
Oct 20 10:39:14 myhost kernel: [ 2187.704685] [ 3320]     0  3320
2041      478   1       0             0 bash
Oct 20 10:39:14 myhost kernel: [ 2187.704687] [ 3436]  1000  3436
63059    15539   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704690] [ 3440]  1000  3440
5483      118   0       0             0 evinced
Oct 20 10:39:14 myhost kernel: [ 2187.704692] [ 3446]  1000  3446
67154    29031   0       0             0 acroread
Oct 20 10:39:14 myhost kernel: [ 2187.704695] [ 3498]  1000  3498
64269    15735   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704698] [ 3506]  1000  3506
1631       66   1       0             0 foxitreader
Oct 20 10:39:14 myhost kernel: [ 2187.704700] [ 3508]  1000  3508
11968     4007   1       0             0 FoxitReader
Oct 20 10:39:14 myhost kernel: [ 2187.704703] [ 3564]  1000  3564
65198    17404   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704705] [ 3622]  1000  3622
62247    14779   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704708] [ 3692]  1000  3692
64126    14386   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704710] [ 3708]  1000  3708
61564    13580   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704713] [ 3770]  1000  3770
63902    16011   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704715] [ 3889]  1000  3889
67617    19788   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704718] [ 3903]  1000  3903
77057    23319   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704720] [ 3927]  1000  3927
60335    14994   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704723] [ 3940]  1000  3940
63049    15434   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704725] [ 3954]  1000  3954
65562    17479   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704727] [ 3972]  1000  3972
64697    16979   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704730] [ 4002]  1000  4002
59109    13305   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704733] [ 4015]  1000  4015
64456    16434   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704735] [ 4028]  1000  4028
63322    15376   0       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704738] [ 4081]  1000  4081
59547    13587   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704740] [ 4085]     0  4085
717       36   0       0             0 sh
Oct 20 10:39:14 myhost kernel: [ 2187.704743] [ 4086]     0  4086
717       36   0       0             0 sh
Oct 20 10:39:14 myhost kernel: [ 2187.704745] [ 4089]     0  4089
717       35   1       0             0 sh
Oct 20 10:39:14 myhost kernel: [ 2187.704748] [ 4092]     0  4092
1185       43   0       0             0 git
Oct 20 10:39:14 myhost kernel: [ 2187.704750] [ 4093]     0  4093
1185       42   1       0             0 git
Oct 20 10:39:14 myhost kernel: [ 2187.704753] [ 4096]     0  4096
717       55   0       0             0 git-pull
Oct 20 10:39:14 myhost kernel: [ 2187.704755] [ 4097]     0  4097
717       54   1       0             0 git-pull
Oct 20 10:39:14 myhost kernel: [ 2187.704758] [ 4128]     0  4128
1187       58   0       0             0 git
Oct 20 10:39:14 myhost kernel: [ 2187.704760] [ 4129]     0  4129
1187       58   1       0             0 git
Oct 20 10:39:14 myhost kernel: [ 2187.704763] [ 4130]     0  4130
1571       86   1       0             0 ssh
Oct 20 10:39:14 myhost kernel: [ 2187.704765] [ 4131]     0  4131
1571       87   0       0             0 ssh
Oct 20 10:39:14 myhost kernel: [ 2187.704767] [ 4141]  1000  4141
65072    15280   1       0             0 evince
Oct 20 10:39:14 myhost kernel: [ 2187.704771] =================
Oct 20 10:39:14 myhost kernel: [ 2187.704773] oom_kill_process:kill task
pid=2069, victim_points=0
Oct 20 10:39:14 myhost kernel: [ 2187.704776]
oom_kill_task:=========kill task pid=2069




here is the page-types log:
             flags	page-count       MB  symbolic-flags
long-symbolic-flags
0x0000000000000000	     17478       68
__________________________________	
0x0000000100000000	      8264       32
______________________r___________	reserved
0x0000000000010000	      2812       10
________________T_________________	compound_tail
0x0000000000008000	        76        0
_______________H__________________	compound_head
0x0000008000000000	         1        0
_____________________________c____	uncached
0x0000000400000001	         3        0
L_______________________d_________	locked,mappedtodisk
0x0000000400000008	         1        0
___U____________________d_________	uptodate,mappedtodisk
0x0000000400000021	        12        0
L____l__________________d_________	locked,lru,mappedtodisk
0x0000000800000024	       718        2
__R__l___________________P________	referenced,lru,private
0x0000000400000028	      2183        8
___U_l__________________d_________	uptodate,lru,mappedtodisk
0x0001000400000028	         4        0
___U_l__________________d_____I___	uptodate,lru,mappedtodisk,readahead
0x000000040000002c	        67        0
__RU_l__________________d_________	referenced,uptodate,lru,mappedtodisk
0x000000000000002c	         4        0
__RU_l____________________________	referenced,uptodate,lru
0x000000080000002c	      2169        8
__RU_l___________________P________	referenced,uptodate,lru,private
0x000000000000402c	      3939       15
__RU_l________b___________________	referenced,uptodate,lru,swapbacked
0x0000000000004038	         1        0
___UDl________b___________________	uptodate,dirty,lru,swapbacked
0x000000000000403c	        71        0
__RUDl________b___________________
referenced,uptodate,dirty,lru,swapbacked
0x000000080000003c	         1        0
__RUDl___________________P________	referenced,uptodate,dirty,lru,private
0x0000000800000060	       107        0
_____lA__________________P________	lru,active,private
0x0000000800000064	       228        0
__R__lA__________________P________	referenced,lru,active,private
0x0000000c00000068	         8        0
___U_lA_________________dP________
uptodate,lru,active,mappedtodisk,private
0x0000000000000068	        16        0
___U_lA___________________________	uptodate,lru,active
0x0000000800000068	       331        1
___U_lA__________________P________	uptodate,lru,active,private
0x0000000400000068	         7        0
___U_lA_________________d_________	uptodate,lru,active,mappedtodisk
0x000000000000006c	        26        0
__RU_lA___________________________	referenced,uptodate,lru,active
0x000000080000006c	        21        0
__RU_lA__________________P________
referenced,uptodate,lru,active,private
0x000000040000006c	       142        0
__RU_lA_________________d_________
referenced,uptodate,lru,active,mappedtodisk
0x0000000000004078	     14103       55
___UDlA_______b___________________	uptodate,dirty,lru,active,swapbacked
0x000000000000407c	      4277       16
__RUDlA_______b___________________
referenced,uptodate,dirty,lru,active,swapbacked
0x0004000000008080	        66        0
_______S_______H________________A_	slab,compound_head,slub_frozen
0x0000000000000080	      2556        9
_______S__________________________	slab
0x0000000000008080	       794        3
_______S_______H__________________	slab,compound_head
0x0004000000000080	        51        0
_______S________________________A_	slab,slub_frozen
0x000000080000012c	         2        0
__RU_l__W________________P________
referenced,uptodate,lru,writeback,private
0x0000000000000400	      1719        6
__________B_______________________	buddy
0x0000000000000800	         1        0
___________M______________________	mmap
0x0000000000000804	         2        0
__R________M______________________	referenced,mmap
0x0000000400000808	         1        0
___U_______M____________d_________	uptodate,mmap,mappedtodisk
0x0000000000000810	         1        0
____D______M______________________	dirty,mmap
0x0000000000008814	         1        0
__R_D______M___H__________________	referenced,dirty,mmap,compound_head
0x0000000000010814	        15        0
__R_D______M____T_________________	referenced,dirty,mmap,compound_tail
0x0000000400000828	       805        3
___U_l_____M____________d_________	uptodate,lru,mmap,mappedtodisk
0x000000040000082c	       467        1
__RU_l_____M____________d_________
referenced,uptodate,lru,mmap,mappedtodisk
0x000000000000482c	         1        0
__RU_l_____M__b___________________
referenced,uptodate,lru,mmap,swapbacked
0x0000000000004838	      5039       19
___UDl_____M__b___________________	uptodate,dirty,lru,mmap,swapbacked
0x000000000000483c	       126        0
__RUDl_____M__b___________________
referenced,uptodate,dirty,lru,mmap,swapbacked
0x000000020004483c	         1        0
__RUDl_____M__b___u____m__________
referenced,uptodate,dirty,lru,mmap,swapbacked,unevictable,mlocked
0x0000000400000868	         9        0
___U_lA____M____________d_________	uptodate,lru,active,mmap,mappedtodisk
0x000000040000086c	      3743       14
__RU_lA____M____________d_________
referenced,uptodate,lru,active,mmap,mappedtodisk
0x0000000c0000086c	         7        0
__RU_lA____M____________dP________
referenced,uptodate,lru,active,mmap,mappedtodisk,private
0x0000000000004878	       581        2
___UDlA____M__b___________________
uptodate,dirty,lru,active,mmap,swapbacked
0x000000000000487c	        89        0
__RUDlA____M__b___________________
referenced,uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000005008	         8        0
___U________a_b___________________	uptodate,anonymous,swapbacked
0x0000000000005808	         4        0
___U_______Ma_b___________________	uptodate,mmap,anonymous,swapbacked
0x0000000000005828	     83024      324
___U_l_____Ma_b___________________
uptodate,lru,mmap,anonymous,swapbacked
0x000000000000582c	        99        0
__RU_l_____Ma_b___________________
referenced,uptodate,lru,mmap,anonymous,swapbacked
0x000000020004582c	         8        0
__RU_l_____Ma_b___u____m__________
referenced,uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x0000000000005838	         2        0
___UDl_____Ma_b___________________
uptodate,dirty,lru,mmap,anonymous,swapbacked
0x0000000000005868	    358737     1401
___U_lA____Ma_b___________________
uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c	        42        0
__RU_lA____Ma_b___________________
referenced,uptodate,lru,active,mmap,anonymous,swapbacked
             total	    515071     2011


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-20  2:58                   ` Figo.zhang
@ 2010-10-20  3:24                     ` KOSAKI Motohiro
  2010-10-20  3:43                       ` Figo.zhang
  0 siblings, 1 reply; 11+ messages in thread
From: KOSAKI Motohiro @ 2010-10-20  3:24 UTC (permalink / raw)
  To: Figo.zhang
  Cc: kosaki.motohiro, Wu Fengguang, KAMEZAWA Hiroyuki, Minchan Kim,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

> 
> > 
> > can you please try 1) invoke oom 2) get page-types -r again. I'm curious
> > that oom makes page accounting lost again. I mean, please send us oom 
> > log and "page-types -r" result.
> > 
> > thanks
> 
> ok, i do the experiment and catch the log:

thanks.


> active_anon:398375 inactive_anon:82967 isolated_anon:0 
>  active_file:81 inactive_file:429 isolated_file:32
>  unevictable:13 dirty:2 writeback:14 unstable:0
>  free:11942 slab_reclaimable:2391 slab_unreclaimable:3303
>  mapped:5617 shmem:33909 pagetables:2280 bounce:0

active_anon + inactive_anon + isolated_anon = 481342 pages ~= 1.8GB
Um, this oom doesn't makes accounting lost.

> here is the page-types log:
>              flags	page-count       MB  symbolic-flags long-symbolic-flags
> 
> 0x0000000000005828	     83024      324 ___U_l_____Ma_b___________________ uptodate,lru,mmap,anonymous,swapbacked
> 0x0000000000005868	    358737     1401 ___U_lA____Ma_b___________________ uptodate,lru,active,mmap,anonymous,swapbacked
>              total	    515071     2011

page-types show similar result.


The big difference is, previous and current are showing some different processes.
only previous has VirtualBox, only current has vmware-usbarbit, etc..

Can you use same test environment?



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-20  3:24                     ` KOSAKI Motohiro
@ 2010-10-20  3:43                       ` Figo.zhang
  2010-10-20  5:05                         ` Adam Jiang
  0 siblings, 1 reply; 11+ messages in thread
From: Figo.zhang @ 2010-10-20  3:43 UTC (permalink / raw)
  To: KOSAKI Motohiro, Wu Fengguang
  Cc: KAMEZAWA Hiroyuki, Minchan Kim, linux-kernel@vger.kernel.org,
	rientjes@google.com, figo1802, linux-mm@kvack.org


> > active_anon:398375 inactive_anon:82967 isolated_anon:0 
> >  active_file:81 inactive_file:429 isolated_file:32
> >  unevictable:13 dirty:2 writeback:14 unstable:0
> >  free:11942 slab_reclaimable:2391 slab_unreclaimable:3303
> >  mapped:5617 shmem:33909 pagetables:2280 bounce:0
> 
> active_anon + inactive_anon + isolated_anon = 481342 pages ~= 1.8GB
> Um, this oom doesn't makes accounting lost.
> 
> > here is the page-types log:
> >              flags	page-count       MB  symbolic-flags long-symbolic-flags
> > 
> > 0x0000000000005828	     83024      324 ___U_l_____Ma_b___________________ uptodate,lru,mmap,anonymous,swapbacked
> > 0x0000000000005868	    358737     1401 ___U_lA____Ma_b___________________ uptodate,lru,active,mmap,anonymous,swapbacked
> >              total	    515071     2011
> 
> page-types show similar result.
> 
> 
> The big difference is, previous and current are showing some different processes.
> only previous has VirtualBox, only current has vmware-usbarbit, etc..
> 
> Can you use same test environment?
yes, it is the same desktop, and i open some pdf files and applications
by random. 

but when my desktop eat up to 1.8GB RAM (active_anon + inactive_anon +
isolated_anon = 481342 pages >= 1.8GB), the system became extraordinary
slow. when i move the mouse, the mouse cant move a little on screen. i
deem it have "crashed", but i ping it's ip by other desktop, it is ok.

so what is apect affect the system seem to "crashed", page-writeback?
page-reclaimed?  and the oom-killer seem to be very conservative? in
that condition , oom_killer must kill some process to release memory for
new process.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: oom_killer crash linux system
  2010-10-20  3:43                       ` Figo.zhang
@ 2010-10-20  5:05                         ` Adam Jiang
  0 siblings, 0 replies; 11+ messages in thread
From: Adam Jiang @ 2010-10-20  5:05 UTC (permalink / raw)
  To: Figo.zhang
  Cc: KOSAKI Motohiro, Wu Fengguang, KAMEZAWA Hiroyuki, Minchan Kim,
	linux-kernel@vger.kernel.org, rientjes@google.com, figo1802,
	linux-mm@kvack.org

On Wed, Oct 20, 2010 at 11:43:45AM +0800, Figo.zhang wrote:
> 
> > > active_anon:398375 inactive_anon:82967 isolated_anon:0 
> > >  active_file:81 inactive_file:429 isolated_file:32
> > >  unevictable:13 dirty:2 writeback:14 unstable:0
> > >  free:11942 slab_reclaimable:2391 slab_unreclaimable:3303
> > >  mapped:5617 shmem:33909 pagetables:2280 bounce:0
> > 
> > active_anon + inactive_anon + isolated_anon = 481342 pages ~= 1.8GB
> > Um, this oom doesn't makes accounting lost.
> > 
> > >              total	    515071     2011
> > 
> > page-types show similar result.
> > 
> > 
> > The big difference is, previous and current are showing some different processes.
> > only previous has VirtualBox, only current has vmware-usbarbit, etc..
> > 
> > Can you use same test environment?
> yes, it is the same desktop, and i open some pdf files and applications
> by random. 
> 
> but when my desktop eat up to 1.8GB RAM (active_anon + inactive_anon +
> isolated_anon = 481342 pages >= 1.8GB), the system became extraordinary
> slow. when i move the mouse, the mouse cant move a little on screen. i
> deem it have "crashed", but i ping it's ip by other desktop, it is ok.
> 
> so what is apect affect the system seem to "crashed", page-writeback?
> page-reclaimed?  and the oom-killer seem to be very conservative? in
> that condition , oom_killer must kill some process to release memory for
> new process.

I think it just simply the test caused system *almost* dead but not
really trigger oom-killer. You have 2GB RAM, right. 0.2G is a huge
amount of memory for Linux kernel.

If you do want to test the new oom-killer, you can just right a simple
program to allocate memory continues but make different instances to
eat memory in different paces. Then, you can find out who will be killed
first eventually.

/Adam
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2010-10-20  5:05 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <20101018021126.GB8654@localhost>
     [not found] ` <1287389631.1997.9.camel@myhost>
     [not found]   ` <20101018180919.3AF8.A69D9226@jp.fujitsu.com>
     [not found]     ` <1287454058.2078.12.camel@myhost>
2010-10-19  2:59       ` oom_killer crash linux system KAMEZAWA Hiroyuki
2010-10-19  5:23         ` Minchan Kim
2010-10-19  5:26           ` KAMEZAWA Hiroyuki
2010-10-19  5:34             ` Minchan Kim
2010-10-20  1:35             ` Wu Fengguang
2010-10-20  2:06               ` Figo.zhang
2010-10-20  2:32                 ` KOSAKI Motohiro
2010-10-20  2:58                   ` Figo.zhang
2010-10-20  3:24                     ` KOSAKI Motohiro
2010-10-20  3:43                       ` Figo.zhang
2010-10-20  5:05                         ` Adam Jiang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).