running of out memory => kernel crash

running of out memory => kernel crash

[Mahmood Naderan]

Hi
I have noticed that when the swap is disabled 

(either via swapoff or not defining any swap partition), 

then running out of memory can cause a kernel panic. 

I wonder why the hungry application won't be killed 

upon a request for a resource that is not available?
In that case, the problematic application will be killed
only but what we see now is that the whole system is
crashed.

 
// Naderan *Mahmood;

Re: running of out memory => kernel crash

[Randy Dunlap]

On Mon, August 8, 2011 11:53 pm, Mahmood Naderan wrote:
> Hi
> I have noticed that when the swap is disabled
>
> (either via swapoff or not defining any swap partition),
>
> then running out of memory can cause a kernel panic. 
>
> I wonder why the hungry application won't be killed
>
> upon a request for a resource that is not available?
> In that case, the problematic application will be killed
> only but what we see now is that the whole system is
> crashed.

Do you have any kernel log panic/oops/Bug messages?

[adding linux-mm mailing list]


-- 
~Randy

Re: running of out memory => kernel crash

[Mahmood Naderan]

>Do you have any kernel log panic/oops/Bug messages?
 
Actually, that happened for one my diskless nodes 10 days ago.
What I saw on the screen (not the logs), was 
"running out of memory.... kernel panic....."

I have not defined any swap partition for diskless nodes. Only the server 
has such partition. 

If you think that is an odd behavior, please let me know what will happen 
exactly when an application needs more memory but there is no room 
left and swap is disabled.
  

// Naderan *Mahmood;


----- Original Message -----
From: Randy Dunlap <rdunlap@xenotime.net>
To: Mahmood Naderan <nt_mahmood@yahoo.com>; linux-mm@kvack.org
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Sent: Tuesday, August 9, 2011 11:36 AM
Subject: Re: running of out memory => kernel crash

On Mon, August 8, 2011 11:53 pm, Mahmood Naderan wrote:
> Hi
> I have noticed that when the swap is disabled
>
> (either via swapoff or not defining any swap partition),
>
> then running out of memory can cause a kernel panic. 
>
> I wonder why the hungry application won't be killed
>
> upon a request for a resource that is not available?
> In that case, the problematic application will be killed
> only but what we see now is that the whole system is
> crashed.

Do you have any kernel log panic/oops/Bug messages?

[adding linux-mm mailing list]


-- 
~Randy

Re: running of out memory => kernel crash

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 923 bytes --]

On Tue, 9 Aug 2011, Mahmood Naderan wrote:

> >Do you have any kernel log panic/oops/Bug messages?
>  
> Actually, that happened for one my diskless nodes 10 days ago.
> What I saw on the screen (not the logs), was 
> "running out of memory.... kernel panic....."
> 

The only similar message in the kernel is "Out of memory and no killable 
processes..." and that panics the machine when there are no eligible 
tasks to kill.

If you're using cpusets or mempolicies, you must ensure that all tasks 
attached to either of them are not set to OOM_DISABLE.  It seems unlikely 
that you're using those, so it seems like a system-wide oom condition.  Do 
cat /proc/*/oom_score and make sure at least some threads have a non-zero 
badness score.  Otherwise, you'll need to adjust their 
/proc/pid/oom_score_adj settings to not be -1000.

Randy also added linux-mm@kvack.org to the cc, but you removed it; please 
don't do that.

Re: running of out memory => kernel crash

[Mahmood Naderan]

>If you're using cpusets or mempolicies, you must ensure that all tasks 
>attached to either of them are not set to OOM_DISABLE.  It seems unlikely 
>that you're using those, so it seems like a system-wide oom condition.
 
I didn't do that manually. What is the default behaviour? Does oom
working or not?

>If you're using cpusets or mempolicies, you must ensure that all tasks 
>attached to either of them are not set to OOM_DISABLE.  It seems unlikely 
>that you're using those, so it seems like a system-wide oom condition.

For a user process:

root@srv:~# cat /proc/18564/oom_score
9198
root@srv:~# cat /proc/18564/oom_adj
0

And for "init" process:

root@srv:~# cat /proc/1/oom_score
17509
root@srv:~# cat /proc/1/oom_adj
0

Based on my understandings, in an out of memory condition (oom),
the init process is more eligible to be killed!!!!!!! Is that right?

Again I didn't get my answer yet:
What is the default behavior of linux in an oom condition? If the default is,
crash (kernel panic), then how can I change that in such a way to kill
the hungry process?

Thanks a lot.

// Naderan *Mahmood;


----- Original Message -----
From: David Rientjes <rientjes@google.com>
To: Mahmood Naderan <nt_mahmood@yahoo.com>
Cc: Randy Dunlap <rdunlap@xenotime.net>; "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>; linux-mm@kvack.org
Sent: Tuesday, August 9, 2011 8:33 PM
Subject: Re: running of out memory => kernel crash

On Tue, 9 Aug 2011, Mahmood Naderan wrote:

> >Do you have any kernel log panic/oops/Bug messages?
>  
> Actually, that happened for one my diskless nodes 10 days ago.
> What I saw on the screen (not the logs), was 
> "running out of memory.... kernel panic....."
> 

The only similar message in the kernel is "Out of memory and no killable 
processes..." and that panics the machine when there are no eligible 
tasks to kill.

If you're using cpusets or mempolicies, you must ensure that all tasks 
attached to either of them are not set to OOM_DISABLE.  It seems unlikely 
that you're using those, so it seems like a system-wide oom condition.  Do 
cat /proc/*/oom_score and make sure at least some threads have a non-zero 
badness score.  Otherwise, you'll need to adjust their 
/proc/pid/oom_score_adj settings to not be -1000.

Randy also added linux-mm@kvack.org to the cc, but you removed it; please 
don't do that.

Re: running of out memory => kernel crash

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1465 bytes --]

On Wed, 10 Aug 2011, Mahmood Naderan wrote:

> >If you're using cpusets or mempolicies, you must ensure that all tasks 
> >attached to either of them are not set to OOM_DISABLE.  It seems unlikely 
> >that you're using those, so it seems like a system-wide oom condition.
>  
> I didn't do that manually. What is the default behaviour? Does oom
> working or not?
> 

The default behavior is to kill all eligible and unkillable threads until 
there are none left to sacrifice (i.e. all kthreads and OOM_DISABLE).

> For a user process:
> 
> root@srv:~# cat /proc/18564/oom_score
> 9198
> root@srv:~# cat /proc/18564/oom_adj
> 0
> 

Ok, so you don't have a /proc/pid/oom_score_adj, so you're using a kernel 
that predates 2.6.36.

> And for "init" process:
> 
> root@srv:~# cat /proc/1/oom_score
> 17509
> root@srv:~# cat /proc/1/oom_adj
> 0
> 
> Based on my understandings, in an out of memory condition (oom),
> the init process is more eligible to be killed!!!!!!! Is that right?
> 

init is exempt from oom killing, it's oom_score is meaningless.

> Again I didn't get my answer yet:
> What is the default behavior of linux in an oom condition? If the default is,
> crash (kernel panic), then how can I change that in such a way to kill
> the hungry process?
> 

You either have /proc/sys/vm/panic_on_oom set or it's killing a thread 
that is taking down the entire machine.  If it's the latter, then please 
capture the kernel log and post it as Randy suggested.

Re: running of out memory => kernel crash

[Mahmood Naderan]

>The default behavior is to kill all eligible and unkillable threads until 
>there are none left to sacrifice (i.e. all kthreads and OOM_DISABLE).
 
In a simple test with virtualbox, I reduced the amount of ram to 300MB. 
Then I ran "swapoff -a" and opened some applications. I noticed that the free
spaces is kept around 2-3MB and "kswapd" is running. Also I saw that disk
activity was very high. 
That mean although "swap" partition is turned off, "kswapd" was trying to do
something. I wonder how that behavior can be explained?

>Ok, so you don't have a /proc/pid/oom_score_adj, so you're using a kernel 
>that predates 2.6.36.
Yes, the srv machine that I posted those results, has kernel before 2.6.36



// Naderan *Mahmood;


----- Original Message -----
From: David Rientjes <rientjes@google.com>
To: Mahmood Naderan <nt_mahmood@yahoo.com>
Cc: Randy Dunlap <rdunlap@xenotime.net>; ""linux-kernel@vger.kernel.org"" <linux-kernel@vger.kernel.org>; "linux-mm@kvack.org" <linux-mm@kvack.org>
Sent: Thursday, August 11, 2011 8:39 AM
Subject: Re: running of out memory => kernel crash

On Wed, 10 Aug 2011, Mahmood Naderan wrote:

> >If you're using cpusets or mempolicies, you must ensure that all tasks 
> >attached to either of them are not set to OOM_DISABLE.  It seems unlikely 
> >that you're using those, so it seems like a system-wide oom condition.
>  
> I didn't do that manually. What is the default behaviour? Does oom
> working or not?
> 

The default behavior is to kill all eligible and unkillable threads until 
there are none left to sacrifice (i.e. all kthreads and OOM_DISABLE).

> For a user process:
> 
> root@srv:~# cat /proc/18564/oom_score
> 9198
> root@srv:~# cat /proc/18564/oom_adj
> 0
> 

Ok, so you don't have a /proc/pid/oom_score_adj, so you're using a kernel 
that predates 2.6.36.

> And for "init" process:
> 
> root@srv:~# cat /proc/1/oom_score
> 17509
> root@srv:~# cat /proc/1/oom_adj
> 0
> 
> Based on my understandings, in an out of memory condition (oom),
> the init process is more eligible to be killed!!!!!!! Is that right?
> 

init is exempt from oom killing, it's oom_score is meaningless.

> Again I didn't get my answer yet:
> What is the default behavior of linux in an oom condition? If the default is,
> crash (kernel panic), then how can I change that in such a way to kill
> the hungry process?
> 

You either have /proc/sys/vm/panic_on_oom set or it's killing a thread 
that is taking down the entire machine.  If it's the latter, then please 
capture the kernel log and post it as Randy suggested.

Re: running of out memory => kernel crash

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 853 bytes --]

On Thu, 11 Aug 2011, Mahmood Naderan wrote:

> >The default behavior is to kill all eligible and unkillable threads until 
> >there are none left to sacrifice (i.e. all kthreads and OOM_DISABLE).
>  
> In a simple test with virtualbox, I reduced the amount of ram to 300MB. 
> Then I ran "swapoff -a" and opened some applications. I noticed that the free
> spaces is kept around 2-3MB and "kswapd" is running. Also I saw that disk
> activity was very high. 
> That mean although "swap" partition is turned off, "kswapd" was trying to do
> something. I wonder how that behavior can be explained?
> 

Despite it's name, kswapd is still active, it's trying to reclaim memory 
to prevent having to kill a process as the last resort.

If /proc/sys/vm/panic_on_oom is not set, as previously mentioned, then 
we'll need the kernel log to diagnose this further.

Re: running of out memory => kernel crash

[Mahmood Naderan]

>Despite it's name, kswapd is still active, it's trying to reclaim memory 
>to prevent having to kill a process as the last resort.
 
I understand what you said, but I did two scenarios:
1- I wrote a simple C++ program that "new" a lot of pointers.
   for ( int i = 0; i < n; i++ ) {
     for ( int j = 0; j < n; j++ ) {
       for ( int k = 0; k < n; k++ ) {
         for ( int l = 0; l < n; l++ ) {
           double *ptr1 = new double[n*i];
           double *ptr2 = new double[n*j];
     }}}}

When I run the program, it ill eat the memory and when it reaches the
maximum ram, it get killed and I saw  message on terminal:

mahmood@vpc:~$ ./leak
Killed

for this scenario, there is no kswapd process running. As it eats the memory
suddenly it get killed.

2- There is 300MB ram. I opened an application saw that
free space reduced to 100MB, then another application reduced the free
space to 30MB. Another application reduced to 4MB. Now the "kswapd"
is running with a lot of disk activity and tries to keep free space at 4MB.
In this scenario, No application is killed.

The question is why in one scenario, the application is killed and in one
scenario, kswapd is running.

I think in the first scenario, the oom_score is calculated more rapidly 
than the second, so immediately is get killed. So kswapd has no chance 
to run because application is killed sooner. In the second scenario, 
kswapd has time to run first. So it will try to free some spaces. However 
since the disk activity is very high, the response time is very high
so the oom_score is calculated lately than first scenario.

Is my understandings correct?


>If /proc/sys/vm/panic_on_oom is not set, as previously mentioned, then 
>we'll need the kernel log to diagnose this further.
I checked that and it is 0. I am trying to reproduce the problem to get some logs


// Naderan *Mahmood;


----- Original Message -----
From: David Rientjes <rientjes@google.com>
To: Mahmood Naderan <nt_mahmood@yahoo.com>
Cc: Randy Dunlap <rdunlap@xenotime.net>; """linux-kernel@vger.kernel.org""" <linux-kernel@vger.kernel.org>; ""linux-mm@kvack.org"" <linux-mm@kvack.org>
Sent: Thursday, August 11, 2011 11:43 AM
Subject: Re: running of out memory => kernel crash

On Thu, 11 Aug 2011, Mahmood Naderan wrote:

> >The default behavior is to kill all eligible and unkillable threads until 
> >there are none left to sacrifice (i.e. all kthreads and OOM_DISABLE).
>  
> In a simple test with virtualbox, I reduced the amount of ram to 300MB. 
> Then I ran "swapoff -a" and opened some applications. I noticed that the free
> spaces is kept around 2-3MB and "kswapd" is running. Also I saw that disk
> activity was very high. 
> That mean although "swap" partition is turned off, "kswapd" was trying to do
> something. I wonder how that behavior can be explained?
> 

Despite it's name, kswapd is still active, it's trying to reclaim memory 
to prevent having to kill a process as the last resort.

If /proc/sys/vm/panic_on_oom is not set, as previously mentioned, then 
we'll need the kernel log to diagnose this further.

Re: running of out memory => kernel crash

[Denys Vlasenko]

On Thu, Aug 11, 2011 at 10:02 AM, Mahmood Naderan <nt_mahmood@yahoo.com> wrote:
>>Despite it's name, kswapd is still active, it's trying to reclaim memory
>>to prevent having to kill a process as the last resort.
>
> I understand what you said, but I did two scenarios:
> 1- I wrote a simple C++ program that "new" a lot of pointers.
>    for ( int i = 0; i < n; i++ ) {
>      for ( int j = 0; j < n; j++ ) {
>        for ( int k = 0; k < n; k++ ) {
>          for ( int l = 0; l < n; l++ ) {
>            double *ptr1 = new double[n*i];
>            double *ptr2 = new double[n*j];
>      }}}}
>
> When I run the program, it ill eat the memory and when it reaches the
> maximum ram, it get killed and I saw  message on terminal:
>
> mahmood@vpc:~$ ./leak
> Killed
>
> for this scenario, there is no kswapd process running.

Why do you think kswapd should get active? What it can possibly do
if there is no swap and therefore it can't free memory by writing
out RAM pages to swap?

> 2- There is 300MB ram. I opened an application saw that
> free space reduced to 100MB, then another application reduced the free
> space to 30MB. Another application reduced to 4MB. Now the "kswapd"
> is running with a lot of disk activity and tries to keep free space at 4MB.
> In this scenario, No application is killed.
>
> The question is why in one scenario, the application is killed and in one
> scenario, kswapd is running.

In scenario 2, the disk activity comes from constant paging in (reading)
of pages which contain code of running binaries.

Since machine has no free RAM and no swap at all, when it needs
a free page it can't swap out a dirty (modified) page or anon
(usually malloced space) page. Thus the only option is to find some
not recently used page with read-only, file-backed content (usually some
binary's text page, but can be any read-only file mapping) and reuse it.

If there are no really old, unused read-only, file-backed pages,
then the discarded page will be needed soon, will need to be read from disk,
and will evict another similar page. Which will be needed soon too,
will need to be read from disk, and will evict another such page...
ad infinitum.

-- 
vda

Re: running of out memory => kernel crash

[Mahmood Naderan]

>What it can possibly do if there is no swap and therefore it 

>can't free memory by writing out RAM pages to swap?


>the disk activity comes from constant paging in (reading)
>of pages which contain code of running binaries.

Why the disk activity does not appear in the first scenario?


>Thus the only option is to find some not recently used page
> with read-only, file-backed content (usually some binary's 

>text page, but can be any read-only file mapping) and reuse it.
Why "killing" does not appear here? Why it try to "find some 

recently used page"?


Both scenarios have one common thing... Running out of memory.
But they behave differently.


// Naderan *Mahmood;


________________________________
From: Denys Vlasenko <vda.linux@googlemail.com>
To: Mahmood Naderan <nt_mahmood@yahoo.com>
Cc: David Rientjes <rientjes@google.com>; Randy Dunlap <rdunlap@xenotime.net>; """"linux-kernel@vger.kernel.org"""" <linux-kernel@vger.kernel.org>; """linux-mm@kvack.org""" <linux-mm@kvack.org>
Sent: Thursday, August 11, 2011 5:17 PM
Subject: Re: running of out memory => kernel crash

On Thu, Aug 11, 2011 at 10:02 AM, Mahmood Naderan <nt_mahmood@yahoo.com> wrote:
>>Despite it's name, kswapd is still active, it's trying to reclaim memory
>>to prevent having to kill a process as the last resort.
>
> I understand what you said, but I did two scenarios:
> 1- I wrote a simple C++ program that "new" a lot of pointers.
>    for ( int i = 0; i < n; i++ ) {
>      for ( int j = 0; j < n; j++ ) {
>        for ( int k = 0; k < n; k++ ) {
>          for ( int l = 0; l < n; l++ ) {
>            double *ptr1 = new double[n*i];
>            double *ptr2 = new double[n*j];
>      }}}}
>
> When I run the program, it ill eat the memory and when it reaches the
> maximum ram, it get killed and I saw  message on terminal:
>
> mahmood@vpc:~$ ./leak
> Killed
>
> for this scenario, there is no kswapd process running.

Why do you think kswapd should get active? What it can possibly do
if there is no swap and therefore it can't free memory by writing
out RAM pages to swap?

> 2- There is 300MB ram. I opened an application saw that
> free space reduced to 100MB, then another application reduced the free
> space to 30MB. Another application reduced to 4MB. Now the "kswapd"
> is running with a lot of disk activity and tries to keep free space at 4MB.
> In this scenario, No application is killed.
>
> The question is why in one scenario, the application is killed and in one
> scenario, kswapd is running.

In scenario 2, the disk activity comes from constant paging in (reading)
of pages which contain code of running binaries.

Since machine has no free RAM and no swap at all, when it needs
a free page it can't swap out a dirty (modified) page or anon
(usually malloced space) page. Thus the only option is to find some
not recently used page with read-only, file-backed content (usually some
binary's text page, but can be any read-only file mapping) and reuse it.

If there are no really old, unused read-only, file-backed pages,
then the discarded page will be needed soon, will need to be read from disk,
and will evict another similar page. Which will be needed soon too,
will need to be read from disk, and will evict another such page...
ad infinitum.

-- 
vda 

Re: running of out memory => kernel crash

[Denys Vlasenko]

On Thursday 11 August 2011 17:13, Mahmood Naderan wrote:
> >What it can possibly do if there is no swap and therefore it 
> 
> >can't free memory by writing out RAM pages to swap?
> 
> 
> >the disk activity comes from constant paging in (reading)
> >of pages which contain code of running binaries.
> 
> Why the disk activity does not appear in the first scenario?

Because there is nowhere to write dirty pages in order to free
some RAM (since you have no swap) and reading in more stuff
from disk can't possibly help with freeing RAM.

(What kernel does in order to free RAM is it drops unmodified
file-backed pages, and doing _that_ doesn't require disk I/O).

Thus, no reading and no writing is necessary/possible.


> >Thus the only option is to find some not recently used page
> > with read-only, file-backed content (usually some binary's 
> 
> >text page, but can be any read-only file mapping) and reuse it.
> Why "killing" does not appear here? Why it try to "find some 
> 
> recently used page"?

Because killing is the last resort. As long as kernel can free
a page by dropping an unmodified file-backed page, it will do that.
When there is nothing more to drop, and still more free pages
are needed, _then_ kernel will start oom killing.

-- 
vda

Re: running of out memory => kernel crash

[Mahmood Naderan]

Thanks a lot Denys for your indepth response

 
// Naderan *Mahmood;


----- Original Message -----
From: Denys Vlasenko <vda.linux@googlemail.com>
To: Mahmood Naderan <nt_mahmood@yahoo.com>
Cc: David Rientjes <rientjes@google.com>; Randy Dunlap <rdunlap@xenotime.net>; ""linux-kernel@vger.kernel.org"" <linux-kernel@vger.kernel.org>; ""linux-mm@kvack.org"" <linux-mm@kvack.org>
Sent: Thursday, August 11, 2011 10:08 PM
Subject: Re: running of out memory => kernel crash

On Thursday 11 August 2011 17:13, Mahmood Naderan wrote:
> >What it can possibly do if there is no swap and therefore it 
> 
> >can't free memory by writing out RAM pages to swap?
> 
> 
> >the disk activity comes from constant paging in (reading)
> >of pages which contain code of running binaries.
> 
> Why the disk activity does not appear in the first scenario?

Because there is nowhere to write dirty pages in order to free
some RAM (since you have no swap) and reading in more stuff
from disk can't possibly help with freeing RAM.

(What kernel does in order to free RAM is it drops unmodified
file-backed pages, and doing _that_ doesn't require disk I/O).

Thus, no reading and no writing is necessary/possible.


> >Thus the only option is to find some not recently used page
> > with read-only, file-backed content (usually some binary's 
> 
> >text page, but can be any read-only file mapping) and reuse it.
> Why "killing" does not appear here? Why it try to "find some 
> 
> recently used page"?

Because killing is the last resort. As long as kernel can free
a page by dropping an unmodified file-backed page, it will do that.
When there is nothing more to drop, and still more free pages
are needed, _then_ kernel will start oom killing.

-- 
vda

Re: running of out memory => kernel crash

[Pavel Ivanov]

Denys,

>> Why "killing" does not appear here? Why it try to "find some
>> recently used page"?
>
> Because killing is the last resort. As long as kernel can free
> a page by dropping an unmodified file-backed page, it will do that.
> When there is nothing more to drop, and still more free pages
> are needed, _then_ kernel will start oom killing.

I have a little concern about this explanation of yours. Suppose we
have some amount of more or less actively executing processes in the
system. Suppose they started to use lots of resident memory. Amount of
memory they use is less than total available physical memory but when
we add total size of code for those processes it would be several
pages more than total size of physical memory. As I understood from
your explanation in such situation one process will execute its time
slice, kernel will switch to other one, find that its code was pushed
out of RAM, read it from disk, execute its time slice, switch to next
process, read its code from disk, execute and so on. So system will be
virtually unusable because of constantly reading from disk just to
execute next small piece of code. But oom will never be firing in such
situation. Is my understanding correct? Shouldn't it be considered as
an unwanted behavior?


Pavel


On Thu, Aug 11, 2011 at 1:38 PM, Denys Vlasenko
<vda.linux@googlemail.com> wrote:
> On Thursday 11 August 2011 17:13, Mahmood Naderan wrote:
>> >What it can possibly do if there is no swap and therefore it
>>
>> >can't free memory by writing out RAM pages to swap?
>>
>>
>> >the disk activity comes from constant paging in (reading)
>> >of pages which contain code of running binaries.
>>
>> Why the disk activity does not appear in the first scenario?
>
> Because there is nowhere to write dirty pages in order to free
> some RAM (since you have no swap) and reading in more stuff
> from disk can't possibly help with freeing RAM.
>
> (What kernel does in order to free RAM is it drops unmodified
> file-backed pages, and doing _that_ doesn't require disk I/O).
>
> Thus, no reading and no writing is necessary/possible.
>
>
>> >Thus the only option is to find some not recently used page
>> > with read-only, file-backed content (usually some binary's
>>
>> >text page, but can be any read-only file mapping) and reuse it.
>> Why "killing" does not appear here? Why it try to "find some
>>
>> recently used page"?
>
> Because killing is the last resort. As long as kernel can free
> a page by dropping an unmodified file-backed page, it will do that.
> When there is nothing more to drop, and still more free pages
> are needed, _then_ kernel will start oom killing.

Re: running of out memory => kernel crash

[Denys Vlasenko]

On Thu, Aug 18, 2011 at 4:18 AM, Pavel Ivanov <paivanof@gmail.com> wrote:
>>> Why "killing" does not appear here? Why it try to "find some
>>> recently used page"?
>>
>> Because killing is the last resort. As long as kernel can free
>> a page by dropping an unmodified file-backed page, it will do that.
>> When there is nothing more to drop, and still more free pages
>> are needed, _then_ kernel will start oom killing.
>
> I have a little concern about this explanation of yours. Suppose we
> have some amount of more or less actively executing processes in the
> system. Suppose they started to use lots of resident memory. Amount of
> memory they use is less than total available physical memory but when
> we add total size of code for those processes it would be several
> pages more than total size of physical memory. As I understood from
> your explanation in such situation one process will execute its time
> slice, kernel will switch to other one, find that its code was pushed
> out of RAM, read it from disk, execute its time slice, switch to next
> process, read its code from disk, execute and so on. So system will be
> virtually unusable because of constantly reading from disk just to
> execute next small piece of code. But oom will never be firing in such
> situation. Is my understanding correct?

Yes.

> Shouldn't it be considered as an unwanted behavior?

Yes. But all alternatives (such as killing some process) seem to be worse.

-- 
vda

Re: running of out memory => kernel crash

[Pavel Ivanov]

On Thu, Aug 18, 2011 at 8:44 AM, Denys Vlasenko
<vda.linux@googlemail.com> wrote:
>> I have a little concern about this explanation of yours. Suppose we
>> have some amount of more or less actively executing processes in the
>> system. Suppose they started to use lots of resident memory. Amount of
>> memory they use is less than total available physical memory but when
>> we add total size of code for those processes it would be several
>> pages more than total size of physical memory. As I understood from
>> your explanation in such situation one process will execute its time
>> slice, kernel will switch to other one, find that its code was pushed
>> out of RAM, read it from disk, execute its time slice, switch to next
>> process, read its code from disk, execute and so on. So system will be
>> virtually unusable because of constantly reading from disk just to
>> execute next small piece of code. But oom will never be firing in such
>> situation. Is my understanding correct?
>
> Yes.
>
>> Shouldn't it be considered as an unwanted behavior?
>
> Yes. But all alternatives (such as killing some process) seem to be worse.

Could you elaborate on this? We have a completely unusable server
which can be revived only by hard power cycling (administrators won't
be able to log in because sshd and shell will fall victims of the same
unending disk reading). And as an alternative we can kill some process
and at least allow administrator to log in and check if something else
can be done to make server feel better. Why is it worse?

I understand that it could be very hard to detect such situation but
at least it's worth trying I think.


Pavel

Re: running of out memory => kernel crash

[Denys Vlasenko]

On Thursday 18 August 2011 16:26, Pavel Ivanov wrote:
> On Thu, Aug 18, 2011 at 8:44 AM, Denys Vlasenko
> <vda.linux@googlemail.com> wrote:
> >> I have a little concern about this explanation of yours. Suppose we
> >> have some amount of more or less actively executing processes in the
> >> system. Suppose they started to use lots of resident memory. Amount of
> >> memory they use is less than total available physical memory but when
> >> we add total size of code for those processes it would be several
> >> pages more than total size of physical memory. As I understood from
> >> your explanation in such situation one process will execute its time
> >> slice, kernel will switch to other one, find that its code was pushed
> >> out of RAM, read it from disk, execute its time slice, switch to next
> >> process, read its code from disk, execute and so on. So system will be
> >> virtually unusable because of constantly reading from disk just to
> >> execute next small piece of code. But oom will never be firing in such
> >> situation. Is my understanding correct?
> >
> > Yes.
> >
> >> Shouldn't it be considered as an unwanted behavior?
> >
> > Yes. But all alternatives (such as killing some process) seem to be worse.
> 
> Could you elaborate on this? We have a completely unusable server
> which can be revived only by hard power cycling (administrators won't
> be able to log in because sshd and shell will fall victims of the same
> unending disk reading).

You can ssh into it. It will just take VERY, VERY LONG.

> And as an alternative we can kill some process 
> and at least allow administrator to log in and check if something else
> can be done to make server feel better. Why is it worse?
> 
> I understand that it could be very hard to detect such situation

Exactly. Server has no means to know when the situation is
bad enough to start killing. IIRC now the rule is simple:
OOM killing starts only when allocations fail.

Perhaps it is possible to add "start OOM killing if less than N free
pages are available", but this will be complex, and won't be good enough
for some configs with many zones (thus, will require even more complications).

-- 
vda

Re: running of out memory => kernel crash

[David Rientjes]

On Fri, 19 Aug 2011, Denys Vlasenko wrote:

> Exactly. Server has no means to know when the situation is
> bad enough to start killing. IIRC now the rule is simple:
> OOM killing starts only when allocations fail.
> 
> Perhaps it is possible to add "start OOM killing if less than N free
> pages are available", but this will be complex, and won't be good enough
> for some configs with many zones (thus, will require even more complications).
> 

Allocations start to fail, and oom killings then start to occur, only when 
the set of allowed zones fall below their minimum watermarks which usually 
includes several hundred kilobytes or several megabytes of free memory for 
lowmem situations.

Re: running of out memory => kernel crash

[Bryan Donlan]

On Thu, Aug 18, 2011 at 10:26, Pavel Ivanov <paivanof@gmail.com> wrote:
> On Thu, Aug 18, 2011 at 8:44 AM, Denys Vlasenko
> <vda.linux@googlemail.com> wrote:
>>> I have a little concern about this explanation of yours. Suppose we
>>> have some amount of more or less actively executing processes in the
>>> system. Suppose they started to use lots of resident memory. Amount of
>>> memory they use is less than total available physical memory but when
>>> we add total size of code for those processes it would be several
>>> pages more than total size of physical memory. As I understood from
>>> your explanation in such situation one process will execute its time
>>> slice, kernel will switch to other one, find that its code was pushed
>>> out of RAM, read it from disk, execute its time slice, switch to next
>>> process, read its code from disk, execute and so on. So system will be
>>> virtually unusable because of constantly reading from disk just to
>>> execute next small piece of code. But oom will never be firing in such
>>> situation. Is my understanding correct?
>>
>> Yes.
>>
>>> Shouldn't it be considered as an unwanted behavior?
>>
>> Yes. But all alternatives (such as killing some process) seem to be worse.
>
> Could you elaborate on this? We have a completely unusable server
> which can be revived only by hard power cycling (administrators won't
> be able to log in because sshd and shell will fall victims of the same
> unending disk reading). And as an alternative we can kill some process
> and at least allow administrator to log in and check if something else
> can be done to make server feel better. Why is it worse?
>
> I understand that it could be very hard to detect such situation but
> at least it's worth trying I think.

Deciding when to call the server unusable is a policy decision that
the kernel can't make very easily on its own; the point when the
system is considered unusable may be different depending on workload.
You could create a userspace daemon, however, that mlockall()s, then
monitors memory usage, load average, etc and kills processes when
things start to go south. You could also use the memory resource
cgroup controller to set hard limits on memory usage.

Re: running of out memory => kernel crash

[Chris Friesen]

On 08/19/2011 01:29 PM, Bryan Donlan wrote:
> On Thu, Aug 18, 2011 at 10:26, Pavel Ivanov<paivanof@gmail.com>  wrote:

>> Could you elaborate on this? We have a completely unusable server
>> which can be revived only by hard power cycling (administrators won't
>> be able to log in because sshd and shell will fall victims of the same
>> unending disk reading). And as an alternative we can kill some process
>> and at least allow administrator to log in and check if something else
>> can be done to make server feel better. Why is it worse?
>>
>> I understand that it could be very hard to detect such situation but
>> at least it's worth trying I think.
>
> Deciding when to call the server unusable is a policy decision that
> the kernel can't make very easily on its own; the point when the
> system is considered unusable may be different depending on workload.
> You could create a userspace daemon, however, that mlockall()s, then
> monitors memory usage, load average, etc and kills processes when
> things start to go south. You could also use the memory resource
> cgroup controller to set hard limits on memory usage.

Indeed.  From the point of view of the OS, it's running everything on 
the system without a problem.  It's deep into swap, but it's running.

If there are application requirements on grade-of-service, it's up to 
the application to check whether those are being met and if not to do 
something about it.

Chris

-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: running of out memory => kernel crash

[Alan Cox]

> Indeed.  From the point of view of the OS, it's running everything on 
> the system without a problem.  It's deep into swap, but it's running.

Watchdogs can help here

> If there are application requirements on grade-of-service, it's up to 
> the application to check whether those are being met and if not to do 
> something about it.

Or it can request such a level of service from the kernel using the
various memory control interfaces provided but not enabled by
distributors in default configurations.

In particular you can tell the kernel to stop the system hitting the
point where it runs near to out of memory + swap and begins to thrash
horribly. For many workloads you will need a lot of pretty much excess
swap, but disk is cheap. It's like banking, you can either pretend it's
safe in which case you do impressions of the US banking system now and
then and the government has to reboot it, or you can do traditional
banking models where you have a reserve which is sufficient to cover the
worst case of making progress. Our zero overcommit isn't specifically
aimed at the page rate problem but is sufficiently related it usually
does the trick.

http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html

I would btw disagree strongly that this is a 'sorry we can't help'
situation. Back when memory was scarce and systems habitually ran at high
memory loads 4.2 and 4.3BSD coped just fine with very high fault rates
that make modern systems curl up and die. That was entirely down to
having good paging and swap policies linked to scheduling behaviour so
they always made progress. Your latency went through the roof but work
got done which meant that if it was transient load the system would feel
like treacle then perk up again where these days it seems the fashion of
most OS's to just explode messily.

In particular they did two things
- Actively tried to swap out all the bits of entire process victims to
  make space to do work under very high load
- When a process was pulled in it got time to run before it as opposed
  to someone else got dumped out

That has two good effects. Firstly the system could write out the process
data very efficiently and get it back likewise. Secondly the system ended
up in a kick one out, do work in the space we have to breath, stop, kick
next out, do work, and in most cases had little CPU contention so could
make good progress in each burst, albeit with the high latency cost.

Alan