out of memory

out of memory

[Ingo Freund]

Hi,

on our database machine with 4 GB RAM and 2 XEON CPUs I got the
following kernel messages.
There are 2 GB RAM declared as shared memory for database usage.
Can anybody explain to me what happened and -may be- why?


Apr  2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0
Apr  2 10:55:09 widbrz01 kernel: Mem-info:
Apr  2 10:55:09 widbrz01 kernel: DMA per-cpu:
Apr  2 10:55:09 widbrz01 kernel: cpu 0 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 0 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 1 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 1 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 2 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 2 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 3 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 3 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: Normal per-cpu:
Apr  2 10:55:09 widbrz01 kernel: cpu 0 hot: low 62, high 186, batch 31 used:174
Apr  2 10:56:10 widbrz01 kernel: cpu 0 cold: low 0, high 62, batch 31 used:52
Apr  2 10:56:10 widbrz01 kernel: cpu 1 hot: low 62, high 186, batch 31 used:86
Apr  2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47
Apr  2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:107
Apr  2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:31
Apr  2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:71
Apr  2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:39
Apr  2 10:56:10 dbm kernel: HighMem per-cpu:
Apr  2 10:56:10 dbm kernel: cpu 0 hot: low 62, high 186, batch 31 used:124
Apr  2 10:56:10 dbm kernel: cpu 0 cold: low 0, high 62, batch 31 used:46
Apr  2 10:56:10 dbm kernel: cpu 1 hot: low 62, high 186, batch 31 used:93
Apr  2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47
Apr  2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:160
Apr  2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:50
Apr  2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:87
Apr  2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:31
Apr  2 10:56:10 dbm kernel: Free pages:        7608kB (868kB HighMem)
Apr  2 10:56:10 dbm kernel: Active:401296 inactive:416067 dirty:76506 writeback:14 unstable:0 free:1902 slab:8550 mapped:399275
pagetables:20137
Apr  2 10:56:10 dbm kernel: DMA free:3572kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:4
all_unreclaimable? yes
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 880 6128
Apr  2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB
pages_scanned:1488 all_unreclaimable? yes
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 41984
Apr  2 10:56:10 dbm kernel: HighMem free:868kB min:512kB low:640kB high:768kB active:1604484kB inactive:1663720kB present:5373952kB
pages_scanned:0 all_unreclaimable? no
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 0
Apr  2 10:56:10 dbm kernel: DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3572kB
Apr  2 10:56:10 dbm kernel: Normal: 88*4kB 4*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3168kB
Apr  2 10:56:10 dbm kernel: HighMem: 5*4kB 4*8kB 3*16kB 2*32kB 3*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 868kB
Apr  2 10:56:10 dbm kernel: Swap cache: add 1, delete 1, find 0/0, race 0+0
Apr  2 10:56:10 dbm kernel: Free swap  = 4209004kB
Apr  2 10:56:10 dbm kernel: Total swap = 4209008kB
Apr  2 10:56:10 dbm kernel: Free swap:       4209004kB
Apr  2 10:56:10 dbm kernel: klogd 1.4.1, ---------- state change ----------
Apr  2 10:56:11 dbm kernel: Inspecting /boot/System.map-2.6.13.2
Apr  2 10:56:11 dbm kernel: Loaded 28714 symbols from /boot/System.map-2.6.13.2.
Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
Apr  2 10:56:11 dbm kernel: No module symbols loaded - kernel modules not enabled.
Apr  2 10:56:11 dbm kernel: 1572864 pages of RAM
Apr  2 10:56:11 dbm kernel: 1343488 pages of HIGHMEM
Apr  2 10:56:11 dbm kernel: 538120 reserved pages
Apr  2 10:56:11 dbm kernel: 10551624 pages shared
Apr  2 10:56:11 dbm kernel: 0 pages swap cached
Apr  2 10:56:11 dbm kernel: 63038 pages dirty
Apr  2 10:56:11 dbm kernel: 14 pages writeback
Apr  2 10:56:11 dbm kernel: 399275 pages mapped
Apr  2 10:56:11 dbm kernel: 8538 pages slab
Apr  2 10:56:11 dbm kernel: 20137 pages pagetables
Apr  2 10:56:11 dbm kernel: Out of Memory: Killed process 18250 (db:SBWI).


Thanks,
Ingo.

Re: out of memory

[Andrew Morton]

Ingo Freund <Ingo.Freund@e-dict.net> wrote:
>
> Hi,
> 
> on our database machine with 4 GB RAM and 2 XEON CPUs I got the
> following kernel messages.
> There are 2 GB RAM declared as shared memory for database usage.
> Can anybody explain to me what happened and -may be- why?
> 

You have a kernel memory leak.

> 
> Apr  2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0

It was a GFP_KERNEL allocation.

> Apr  2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB

There's a grand total of 1.2MB of ZONE_NORMAL memory on the LRU.  The rest
(900MB-odd) is lost.

> Apr  2 10:56:11 dbm kernel: 8538 pages slab

and it's not in slab.

> Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.

Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then,
but I do not recall any particular patch which might fix this, sorry.

Your best option would be to seek a kernel upgrade.

Re: out of memory

[Ingo Freund]

Andrew Morton wrote:
> Ingo Freund <Ingo.Freund@e-dict.net> wrote:
>> Apr  2 10:56:11 dbm kernel: 8538 pages slab
> 
> and it's not in slab.
> 
>> Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
> 
> Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then,
> but I do not recall any particular patch which might fix this, sorry.
> 
> Your best option would be to seek a kernel upgrade.

I'll give a try to the last kernel version

Is there a way to get those kernel/memory information from time
to time from the running system which I found in the syslog file?

Thank you
Ingo.

Re: out of memory

[Andrew Morton]

Ingo Freund <Ingo.Freund@e-dict.net> wrote:
>

Please don't edit Cc:'s when working with kernel developers - just do
reply-to-all.

> Andrew Morton wrote:
> > Ingo Freund <Ingo.Freund@e-dict.net> wrote:
> >> Apr  2 10:56:11 dbm kernel: 8538 pages slab
> > 
> > and it's not in slab.
> > 
> >> Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
> > 
> > Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then,
> > but I do not recall any particular patch which might fix this, sorry.
> > 
> > Your best option would be to seek a kernel upgrade.
> 
> I'll give a try to the last kernel version

OK, 2.6.16.1 would suit.

> Is there a way to get those kernel/memory information from time
> to time from the running system which I found in the syslog file?

/proc/meminfo gives a decent summary.

To watch your ZONE_NORMAL disappearing it would be better to monitor
/proc/zoneinfo.  There you'll see `free', `active', `inactive' and
`present'.  In /proc/meminfo you'll find the slab utilisation.

It should be approximately true that free+active+inactive+slab = present.

out of memory

[Balram Adlakha]

Why does my system logs have:
kernel : out of memory, killed process (xyz)

I have 256mb ram, and a huge swap partition which never gets used!
I am currently on 2.5.60 but i've been having this problem for a long time...