Overcommit (OOM) problem on embedded device (PPChameleon)

Overcommit (OOM) problem on embedded device (PPChameleon)

[Martin Egholm Nielsen]

Hi there,

I hope this is the place to go...

I have a some problems figuring out the OOM-killer and configuring the
overcommit_memory parameter. Hope someone here can guide me in the right
directions...

Specs:
I'm having an embedded Linux system running on a PPC405EP (PPChameleon)
with 64 megs of RAM, some flash, but (ofcourse) no swap space. It runs a
2.4.20 kernel patched with drivers for my device.

Problem:
I have an application that is killed by the OOM (I guess) when it tries
to "use" more memory than present on the system.
Bolied down, memory is allocated with "sbrk" and then touch'ed (see
test-application below).

With "/proc/sys/vm/overcommit_memory" set to 2, I expected that "sbrk"
would return "-1L" (0xFFFFFFFF), but it doesn't, hence is
terminated/killed by the kernel.

However, both my desktop Linux (RH 7.3)/2.4.18-10/i386 and Linux
(FC2)/2.6.5/i386 did what I expected:

# ./exhaust_mem
...
ffffffff

Out of memory
# #Yeaaaah!

Having searched the web, I see that this may be related with the fact
that there is no swap enabled on the embedded device.
However, I tried disabling the swap (commented in fstab), but the
desktop linux still behaves "correct".

Can I do anything in order to get it the way I expected?

Best regards,
  Martin Egholm

=== exhaust_mem.c ===

#include <unistd.h>
#include <stdio.h>
#define SIZE 1000000

int main( int i )
{
   while ( 1 ) {
     char *v = sbrk( SIZE );
     char *p;

     printf( "%x\n\n", v );

     if ((long)v < 0) {
       fprintf(stderr, "Out of memory\n");
       exit(1);
     } // if

     for (p = v; p < v + SIZE; ++p) {
       *p = 42;
     } // for

   } // while
} // main

Re: Overcommit (OOM) problem on embedded device (PPChameleon)

[Eugene Surovegin]

On Tue, Mar 29, 2005 at 09:30:57PM +0200, Martin Egholm Nielsen wrote:

[snip]

> Specs:
> I'm having an embedded Linux system running on a PPC405EP (PPChameleon)
> with 64 megs of RAM, some flash, but (ofcourse) no swap space. It runs a
> 2.4.20 kernel patched with drivers for my device.
> 
> Problem:
> I have an application that is killed by the OOM (I guess) when it tries
> to "use" more memory than present on the system.
> Bolied down, memory is allocated with "sbrk" and then touch'ed (see
> test-application below).

[snip]

> However, both my desktop Linux (RH 7.3)/2.4.18-10/i386 and Linux
> (FC2)/2.6.5/i386 did what I expected:

Well, the only way to get the same results is to use the _same_ linux 
kernel version and have the _same_ hw configuration (e.g. memory 
configuration).

There were quite a few changes to VM during 2.4 lifetime; 2.6 is 
different from 2.4 in the memory management area. RH kernel is 
probably patched as hell, so who knows what VM it has.

--
Eugene

RE: Overcommit (OOM) problem on embedded device (PPChameleon)

[Per Hallsmark]

I think your main problem is that the stock 2.4.x kernel doesn't support
ovecommit_memory type 2 (i believe)...
Look in mm/mmap.c and search for overcommit_memory, then do same
in the sources for your redhat kernels.

Perhaps there's a patch for it floating around somewhere? ;-)

>===== Original Message From Martin Egholm Nielsen <martin@egholm-nielsen.dk> 
=====
>Hi there,
>
>I hope this is the place to go...
>
>I have a some problems figuring out the OOM-killer and configuring the
>overcommit_memory parameter. Hope someone here can guide me in the right
>directions...
>
>Specs:
>I'm having an embedded Linux system running on a PPC405EP (PPChameleon)
>with 64 megs of RAM, some flash, but (ofcourse) no swap space. It runs a
>2.4.20 kernel patched with drivers for my device.
>
>Problem:
>I have an application that is killed by the OOM (I guess) when it tries
>to "use" more memory than present on the system.
>Bolied down, memory is allocated with "sbrk" and then touch'ed (see
>test-application below).
>
>With "/proc/sys/vm/overcommit_memory" set to 2, I expected that "sbrk"
>would return "-1L" (0xFFFFFFFF), but it doesn't, hence is
>terminated/killed by the kernel.
>
>However, both my desktop Linux (RH 7.3)/2.4.18-10/i386 and Linux
>(FC2)/2.6.5/i386 did what I expected:
>
># ./exhaust_mem
>...
>ffffffff
>
>Out of memory
># #Yeaaaah!
>
>Having searched the web, I see that this may be related with the fact
>that there is no swap enabled on the embedded device.
>However, I tried disabling the swap (commented in fstab), but the
>desktop linux still behaves "correct".
>
>Can I do anything in order to get it the way I expected?
>
>Best regards,
>  Martin Egholm
>
>=== exhaust_mem.c ===
>
>#include <unistd.h>
>#include <stdio.h>
>#define SIZE 1000000
>
>int main( int i )
>{
>   while ( 1 ) {
>     char *v = sbrk( SIZE );
>     char *p;
>
>     printf( "%x\n\n", v );
>
>     if ((long)v < 0) {
>       fprintf(stderr, "Out of memory\n");
>       exit(1);
>     } // if
>
>     for (p = v; p < v + SIZE; ++p) {
>       *p = 42;
>     } // for
>
>   } // while
>} // main
>
>
>
>
>
>
>_______________________________________________
>Linuxppc-embedded mailing list
>Linuxppc-embedded@ozlabs.org
>https://ozlabs.org/mailman/listinfo/linuxppc-embedded

Re: Overcommit (OOM) problem on embedded device (PPChameleon)

[Martin Egholm Nielsen]

Hi Per,

> I think your main problem is that the stock 2.4.x kernel doesn't support
> ovecommit_memory type 2 (i believe)...
Doooh! You're so right!
I read the docs on overcommit_memory from 2.6 - stupid me!

> Look in mm/mmap.c and search for overcommit_memory, then do same
> in the sources for your redhat kernels.
I'll look into RH kernel to see if it isn't like you say :-)

> Perhaps there's a patch for it floating around somewhere? ;-)
Now, that would be really nice.
Though I have no idea of where to look!?

Thanks for clearing my mind!

BR,
  Martin

> 
> 
>>===== Original Message From Martin Egholm Nielsen <martin@egholm-nielsen.dk> 
> 
> =====
> 
>>Hi there,
>>
>>I hope this is the place to go...
>>
>>I have a some problems figuring out the OOM-killer and configuring the
>>overcommit_memory parameter. Hope someone here can guide me in the right
>>directions...
>>
>>Specs:
>>I'm having an embedded Linux system running on a PPC405EP (PPChameleon)
>>with 64 megs of RAM, some flash, but (ofcourse) no swap space. It runs a
>>2.4.20 kernel patched with drivers for my device.
>>
>>Problem:
>>I have an application that is killed by the OOM (I guess) when it tries
>>to "use" more memory than present on the system.
>>Bolied down, memory is allocated with "sbrk" and then touch'ed (see
>>test-application below).
>>
>>With "/proc/sys/vm/overcommit_memory" set to 2, I expected that "sbrk"
>>would return "-1L" (0xFFFFFFFF), but it doesn't, hence is
>>terminated/killed by the kernel.
>>
>>However, both my desktop Linux (RH 7.3)/2.4.18-10/i386 and Linux
>>(FC2)/2.6.5/i386 did what I expected:
>>
>># ./exhaust_mem
>>...
>>ffffffff
>>
>>Out of memory
>># #Yeaaaah!
>>
>>Having searched the web, I see that this may be related with the fact
>>that there is no swap enabled on the embedded device.
>>However, I tried disabling the swap (commented in fstab), but the
>>desktop linux still behaves "correct".
>>
>>Can I do anything in order to get it the way I expected?
>>
>>Best regards,
>> Martin Egholm
>>
>>=== exhaust_mem.c ===
>>
>>#include <unistd.h>
>>#include <stdio.h>
>>#define SIZE 1000000
>>
>>int main( int i )
>>{
>>  while ( 1 ) {
>>    char *v = sbrk( SIZE );
>>    char *p;
>>
>>    printf( "%x\n\n", v );
>>
>>    if ((long)v < 0) {
>>      fprintf(stderr, "Out of memory\n");
>>      exit(1);
>>    } // if
>>
>>    for (p = v; p < v + SIZE; ++p) {
>>      *p = 42;
>>    } // for
>>
>>  } // while
>>} // main
>>
>>
>>
>>
>>
>>
>>_______________________________________________
>>Linuxppc-embedded mailing list
>>Linuxppc-embedded@ozlabs.org
>>https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> 
> 
> 
> 
> 

Re: Overcommit (OOM) problem on embedded device (PPChameleon)

[Martin Egholm Nielsen]

Hi Per,

> I think your main problem is that the stock 2.4.x kernel doesn't support
> ovecommit_memory type 2 (i believe)...
Doooh! You're so right!
I read the docs on overcommit_memory from 2.6 - stupid me!

> Look in mm/mmap.c and search for overcommit_memory, then do same
> in the sources for your redhat kernels.
I'll look into RH kernel to see if it isn't like you say :-)

> Perhaps there's a patch for it floating around somewhere? ;-)
Now, that would be really nice.
Though I have no idea of where to look!?

Thanks for clearing my mind!

BR,
  Martin

> 
> 
>>===== Original Message From Martin Egholm Nielsen <martin@egholm-nielsen.dk> 
> 
> =====
> 
>>Hi there,
>>
>>I hope this is the place to go...
>>
>>I have a some problems figuring out the OOM-killer and configuring the
>>overcommit_memory parameter. Hope someone here can guide me in the right
>>directions...
>>
>>Specs:
>>I'm having an embedded Linux system running on a PPC405EP (PPChameleon)
>>with 64 megs of RAM, some flash, but (ofcourse) no swap space. It runs a
>>2.4.20 kernel patched with drivers for my device.
>>
>>Problem:
>>I have an application that is killed by the OOM (I guess) when it tries
>>to "use" more memory than present on the system.
>>Bolied down, memory is allocated with "sbrk" and then touch'ed (see
>>test-application below).
>>
>>With "/proc/sys/vm/overcommit_memory" set to 2, I expected that "sbrk"
>>would return "-1L" (0xFFFFFFFF), but it doesn't, hence is
>>terminated/killed by the kernel.
>>
>>However, both my desktop Linux (RH 7.3)/2.4.18-10/i386 and Linux
>>(FC2)/2.6.5/i386 did what I expected:
>>
>># ./exhaust_mem
>>...
>>ffffffff
>>
>>Out of memory
>># #Yeaaaah!
>>
>>Having searched the web, I see that this may be related with the fact
>>that there is no swap enabled on the embedded device.
>>However, I tried disabling the swap (commented in fstab), but the
>>desktop linux still behaves "correct".
>>
>>Can I do anything in order to get it the way I expected?
>>
>>Best regards,
>> Martin Egholm
>>
>>=== exhaust_mem.c ===
>>
>>#include <unistd.h>
>>#include <stdio.h>
>>#define SIZE 1000000
>>
>>int main( int i )
>>{
>>  while ( 1 ) {
>>    char *v = sbrk( SIZE );
>>    char *p;
>>
>>    printf( "%x\n\n", v );
>>
>>    if ((long)v < 0) {
>>      fprintf(stderr, "Out of memory\n");
>>      exit(1);
>>    } // if
>>
>>    for (p = v; p < v + SIZE; ++p) {
>>      *p = 42;
>>    } // for
>>
>>  } // while
>>} // main
>>
>>
>>
>>
>>
>>
>>_______________________________________________
>>Linuxppc-embedded mailing list
>>Linuxppc-embedded@ozlabs.org
>>https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> 
> 
> 
> 
> 

RE: Overcommit (OOM) problem on embedded device (PPChameleon)

[David Adair]

> > Look in mm/mmap.c and search for overcommit_memory, then do same
> > in the sources for your redhat kernels.
> I'll look into RH kernel to see if it isn't like you say :-)
> 
> > Perhaps there's a patch for it floating around somewhere? ;-)
> Now, that would be really nice.
> Though I have no idea of where to look!?
> 

Here is what I do.  Patch is against BK linuxppc-2.4 version 1.1285
(2.4.28-pre3) but might work.

Burns 5% of the RAM but alternatives all seem worse and I never
have much luck trying to convince my colleagues that
dynamic allocation has no place in an embedded system.

If you find out why RH works it would be interesting ... perhaps
there is a better way.

David


#
# mm/mmap.c
#   Add pessimistic overcommit mode similar to 2.6 mode 2.
#   This allows malloc aka sbrk() to actually fail before
#   process is killed.
#   Overloaded sysctl_overcommit_memory to be both enable
#   and ratio to avoid making a new sysctl.
# 
diff -Nru a/mm/mmap.c b/mm/mmap.c
--- a/mm/mmap.c	2005-03-30 07:15:13 -08:00
+++ b/mm/mmap.c	2005-03-30 07:15:13 -08:00
@@ -45,9 +45,13 @@
 	__S000, __S001, __S010, __S011, __S100, __S101, __S110, __S111
 };
 
-int sysctl_overcommit_memory;
+int sysctl_overcommit_memory = 98;
+
 int max_map_count = DEFAULT_MAX_MAP_COUNT;
 
+extern unsigned long totalram_pages;
+extern unsigned long totalhigh_pages;
+
 /* Check that a process has enough memory to allocate a
  * new virtual mapping.
  */
@@ -66,7 +70,7 @@
 	unsigned long free;
 	
         /* Sometimes we want to use more memory than we have. */
-	if (sysctl_overcommit_memory)
+	if (sysctl_overcommit_memory == 1)
 	    return 1;
 
 	/* The page cache contains buffer pages these days.. */
@@ -91,7 +95,20 @@
 	free += (dentry_stat.nr_unused * sizeof(struct dentry)) >>
PAGE_SHIFT;
 	free += (inodes_stat.nr_unused * sizeof(struct inode)) >>
PAGE_SHIFT;
 
+	/*
+	 * Leave the last 3% for root
+	 */
+	if (current->euid)
+		free -= free / 32;
+
+	/* Strict mode do not allocate last bit of memory */
+	if (sysctl_overcommit_memory) {
+		pages += (totalram_pages - totalhigh_pages)
+			* (100 - sysctl_overcommit_memory) / 100;
+	}
+
 	return free > pages;
+
 }
 
 /* Remove one vm structure from the inode's i_mapping address space. */

Re: Overcommit (OOM) problem on embedded device (PPChameleon)

[Martin Egholm Nielsen]

Hi David,

>>>Look in mm/mmap.c and search for overcommit_memory, then do same
>>>in the sources for your redhat kernels.
>>I'll look into RH kernel to see if it isn't like you say :-)
>>>Perhaps there's a patch for it floating around somewhere? ;-)
>>Now, that would be really nice.
>>Though I have no idea of where to look!?
> Here is what I do.  Patch is against BK linuxppc-2.4 version 1.1285
> (2.4.28-pre3) but might work.
Well, the patch goes straight into my 2.4.20 ppc-kernel - but we have to 
wait until tomorrow to see if it does the magic.

> Burns 5% of the RAM but alternatives all seem worse and I never
I can live with that (3%)...

> If you find out why RH works it would be interesting ... perhaps
> there is a better way.
I downloaded the 2.4.20-10 kernel SRPM from RH8 - it contains all the 
patches they apply to the serene 2.4.20 kernel.
There is some stuff regarding introducing the 
"overcommit_memory_accounting" variable from the 2.5 kernel, but I 
haven't looked into it in depth.

I'll try with your patch first :-)

BR,
  Martin Egholm

> #
> # mm/mmap.c
> #   Add pessimistic overcommit mode similar to 2.6 mode 2.
> #   This allows malloc aka sbrk() to actually fail before
> #   process is killed.
> #   Overloaded sysctl_overcommit_memory to be both enable
> #   and ratio to avoid making a new sysctl.
> # 
> diff -Nru a/mm/mmap.c b/mm/mmap.c
> --- a/mm/mmap.c	2005-03-30 07:15:13 -08:00
> +++ b/mm/mmap.c	2005-03-30 07:15:13 -08:00
> @@ -45,9 +45,13 @@
>  	__S000, __S001, __S010, __S011, __S100, __S101, __S110, __S111
>  };
>  
> -int sysctl_overcommit_memory;
> +int sysctl_overcommit_memory = 98;
> +
>  int max_map_count = DEFAULT_MAX_MAP_COUNT;
>  
> +extern unsigned long totalram_pages;
> +extern unsigned long totalhigh_pages;
> +
>  /* Check that a process has enough memory to allocate a
>   * new virtual mapping.
>   */
> @@ -66,7 +70,7 @@
>  	unsigned long free;
>  	
>          /* Sometimes we want to use more memory than we have. */
> -	if (sysctl_overcommit_memory)
> +	if (sysctl_overcommit_memory == 1)
>  	    return 1;
>  
>  	/* The page cache contains buffer pages these days.. */
> @@ -91,7 +95,20 @@
>  	free += (dentry_stat.nr_unused * sizeof(struct dentry)) >>
> PAGE_SHIFT;
>  	free += (inodes_stat.nr_unused * sizeof(struct inode)) >>
> PAGE_SHIFT;
>  
> +	/*
> +	 * Leave the last 3% for root
> +	 */
> +	if (current->euid)
> +		free -= free / 32;
> +
> +	/* Strict mode do not allocate last bit of memory */
> +	if (sysctl_overcommit_memory) {
> +		pages += (totalram_pages - totalhigh_pages)
> +			* (100 - sysctl_overcommit_memory) / 100;
> +	}
> +
>  	return free > pages;
> +
>  }
>  
>  /* Remove one vm structure from the inode's i_mapping address space. */
> 
> 
> 

Re: Overcommit (OOM) problem on embedded device (PPChameleon)

[Martin Egholm Nielsen]

Helloooo David!

>>>Look in mm/mmap.c and search for overcommit_memory, then do same
>>>in the sources for your redhat kernels.
>>I'll look into RH kernel to see if it isn't like you say :-)
>>>Perhaps there's a patch for it floating around somewhere? ;-)
>>Now, that would be really nice.
>>Though I have no idea of where to look!?
> Here is what I do.  Patch is against BK linuxppc-2.4 version 1.1285
> (2.4.28-pre3) but might work.
Uhhuuuuuuu!! It works!!!

With my simple exhaust_mem example it works with a value of 98 (that is 
5% of waste).
However, I need setting it to 95 (so ~8% waste) in order to make it work 
entirely with my other application... But that's better than letting the 
OOM take over control.

> Burns 5% of the RAM but alternatives all seem worse and I never
> have much luck trying to convince my colleagues that
> dynamic allocation has no place in an embedded system.
> 
> If you find out why RH works it would be interesting ... perhaps
> there is a better way.
I'll let you know if I find the triggering patch in RH's kernels :-)

THANKS,
  Martin Egholm

> 
> David
> 
> 
> #
> # mm/mmap.c
> #   Add pessimistic overcommit mode similar to 2.6 mode 2.
> #   This allows malloc aka sbrk() to actually fail before
> #   process is killed.
> #   Overloaded sysctl_overcommit_memory to be both enable
> #   and ratio to avoid making a new sysctl.
> # 
> diff -Nru a/mm/mmap.c b/mm/mmap.c
> --- a/mm/mmap.c	2005-03-30 07:15:13 -08:00
> +++ b/mm/mmap.c	2005-03-30 07:15:13 -08:00
> @@ -45,9 +45,13 @@
>  	__S000, __S001, __S010, __S011, __S100, __S101, __S110, __S111
>  };
>  
> -int sysctl_overcommit_memory;
> +int sysctl_overcommit_memory = 98;
> +
>  int max_map_count = DEFAULT_MAX_MAP_COUNT;
>  
> +extern unsigned long totalram_pages;
> +extern unsigned long totalhigh_pages;
> +
>  /* Check that a process has enough memory to allocate a
>   * new virtual mapping.
>   */
> @@ -66,7 +70,7 @@
>  	unsigned long free;
>  	
>          /* Sometimes we want to use more memory than we have. */
> -	if (sysctl_overcommit_memory)
> +	if (sysctl_overcommit_memory == 1)
>  	    return 1;
>  
>  	/* The page cache contains buffer pages these days.. */
> @@ -91,7 +95,20 @@
>  	free += (dentry_stat.nr_unused * sizeof(struct dentry)) >>
> PAGE_SHIFT;
>  	free += (inodes_stat.nr_unused * sizeof(struct inode)) >>
> PAGE_SHIFT;
>  
> +	/*
> +	 * Leave the last 3% for root
> +	 */
> +	if (current->euid)
> +		free -= free / 32;
> +
> +	/* Strict mode do not allocate last bit of memory */
> +	if (sysctl_overcommit_memory) {
> +		pages += (totalram_pages - totalhigh_pages)
> +			* (100 - sysctl_overcommit_memory) / 100;
> +	}
> +
>  	return free > pages;
> +
>  }
>  
>  /* Remove one vm structure from the inode's i_mapping address space. */
> 
> 
>