Out of memory and no killable processes: 2.6.22-2-686-bigmem

Out of memory and no killable processes: 2.6.22-2-686-bigmem

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

Hello!

We are running Debian with 2.6.22-2-686-bigmem on Dell Blade 1955 hardware
and get a Kernel Panic with oom + message that there are no processes
left to kill:

http://home.schottelius.org/~nico/unix/linux/oom_no_killable-2.6.22-1.jpeg

Anyone an idea, what's the cause for that? This error happened on two of
those machines,

What I can see in our analysis done with munin is that the number of
open inodes and inode table size decreased within some days from 40k
to next to zero. Munin uses

   awk '{print "used.value " $1-$2 "\nmax.value " $1}' < /proc/sys/fs/inode-nr

to log those value (happened on both machines).

Thanks for any hint and CC as usual, please.

Nico

-- 
Think about Free and Open Source Software (FOSS).
http://nico.schottelius.org/documentations/foss/the-term-foss/

PGP: BFE4 C736 ABE5 406F 8F42  F7CF B8BE F92A 9885 188C

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Out of memory and no killable processes: 2.6.22-2-686-bigmem

[Robert Hancock]

Nico Schottelius wrote:
> Hello!
> 
> We are running Debian with 2.6.22-2-686-bigmem on Dell Blade 1955 hardware
> and get a Kernel Panic with oom + message that there are no processes
> left to kill:
> 
> http://home.schottelius.org/~nico/unix/linux/oom_no_killable-2.6.22-1.jpeg
> 
> Anyone an idea, what's the cause for that? This error happened on two of
> those machines,
> 
> What I can see in our analysis done with munin is that the number of
> open inodes and inode table size decreased within some days from 40k
> to next to zero. Munin uses
> 
>    awk '{print "used.value " $1-$2 "\nmax.value " $1}' < /proc/sys/fs/inode-nr
> 
> to log those value (happened on both machines).
> 
> Thanks for any hint and CC as usual, please.

How much RAM is in these machines? If you're running tons of memory, it 
really is better to run a 64-bit kernel if possible. I believe there are 
some cases where low memory can be pretty easily exhausted on machines 
with lots of high memory.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Out of memory and no killable processes: 2.6.22-2-686-bigmem

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

Hello Robert,

Robert Hancock [Tue, Dec 18, 2007 at 06:43:22PM -0600]:
> How much RAM is in these machines?

8 GiB

> If you're running tons of memory, it 
> really is better to run a 64-bit kernel if possible.

Sure? Afaik that results in a bit slower access to memory and appart
from being able to address MUCH more memory doesn't change the
situation.

Perhaps I missed something.

And updating to 64 Bit kernel also implies changing the userland, so
a reinstall would be necessary, wouldn't it?

> I believe there are 
> some cases where low memory can be pretty easily exhausted on machines with 
> lots of high memory.

Some minutes ago the third machine of that type freezed.
with 7 GiB of *free* memory, before getting oom+panic.

The thing that looks pretty the same is that the inode table / open
inodes shrinked very heavy from 41k to about 2k before the freeze
in about 3 days.

Same kernel, same Dell 1955 hardware, different machine.

Is it possible to debug this issue somehow?

Nico

-- 
Think about Free and Open Source Software (FOSS).
http://nico.schottelius.org/documentations/foss/the-term-foss/

PGP: BFE4 C736 ABE5 406F 8F42  F7CF B8BE F92A 9885 188C

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Out of memory and no killable processes: 2.6.22-2-686-bigmem

[Parag Warudkar]

On Wed, 19 Dec 2007, Nico Schottelius wrote:

>> If you're running tons of memory, it
>> really is better to run a 64-bit kernel if possible.
>
> Sure? Afaik that results in a bit slower access to memory and appart
> from being able to address MUCH more memory doesn't change the
> situation.

No, generally 64-bit x86 does not slow down things - it might speed things 
up a bit due to more accessible registers in 64-bit mode. And it does 
change the situation from VM PoV a lot. In (bearded) God's own words - 
"... run a 64bit kernel and then the kernel isn't trying the software equivalent of 
managing a filing cabinet through the keyhole."

You will find this recent thread on same topic useful - 
http://lkml.org/lkml/2007/12/17/7 .


> Perhaps I missed something.
>
> And updating to 64 Bit kernel also implies changing the userland, so
> a reinstall would be necessary, wouldn't it?

May not be necessary - I recently tried installing Fedora 8 32-bit/PAE on 
a 8Gb Core 2 box and then upgrading just the kernel to 64-bit and it worked 
fine except for the 32-bit iptables service having problems running on 
64-bit kernel. [Also I hear the closed source nVidia binaries will not 
work on this setup - so if you don't need such things you will be able to 
go to a 64-bit kernel w/o reinstalling userspace.]

>> I believe there are
>> some cases where low memory can be pretty easily exhausted on machines with
>> lots of high memory.
>
> Some minutes ago the third machine of that type freezed.
> with 7 GiB of *free* memory, before getting oom+panic.
>
> Is it possible to debug this issue somehow?

Like said in the above thread I referred to you might have better luck 
with RHEL/CentOS or other "Enterprise" kernels if you want to stick to 
32-bit. But the pain is not worthwhile - give 64-bit kernel a shot.

HTH

Parag

Re: Out of memory and no killable processes: 2.6.22-2-686-bigmem

[Jiri Slaby]

On 12/19/2007 06:55 AM, Nico Schottelius wrote:
> And updating to 64 Bit kernel also implies changing the userland, so
> a reinstall would be necessary, wouldn't it?

No, if you enable ia32 support.

Re: Out of memory and no killable processes: 2.6.22-2-686-bigmem

[Jiri Slaby]

On 12/19/2007 09:56 AM, Parag Warudkar wrote:
> running on 64-bit kernel. [Also I hear the closed source nVidia binaries
> will not work on this setup - so if you don't need such things you will
> be able to go to a 64-bit kernel w/o reinstalling userspace.]

False, I'm running 100-* nvidia drivers on such set up system. However some
drivers might be broken due to unimplemented (not filled) compat_ioctl.