oom-killings, but I'm not out of memory!

oom-killings, but I'm not out of memory!

[Anthony DiSante]

Hello,

I'm running a 2.6.11 kernel.  I have 1 gig of RAM and 1 gig of swap.  Lately 
when my RAM gets full, the oom-killer takes out either Mozilla or 
Thunderbird (my two biggest memory hogs), even though my swap space is only 
20% full.  I still have ~800 MB of free swap space, so shouldn't the kernel 
push Moz or T-bird into swap instead of oom-killing it?  At their maximum 
memory-hogging capacity, neither Moz nor T-bird is ever using more than 200 MB.

Thanks,
Anthony DiSante
http://nodivisions.com/


Jun 28 12:09:09 soma oom-killer: gfp_mask=0x80d2
...
Jun 28 12:09:09 soma Free swap  = 781012kB
Jun 28 12:09:09 soma Total swap = 987988kB
Jun 28 12:09:09 soma Out of Memory: Killed process 30787 (thunderbird-bin).
Jun 28 12:09:09 soma Out of Memory: Killed process 18112 (thunderbird-bin).
Jun 28 12:09:09 soma Out of Memory: Killed process 18116 (thunderbird-bin).
Jun 28 12:09:09 soma Out of Memory: Killed process 18117 (thunderbird-bin).
Jun 28 12:09:09 soma Out of Memory: Killed process 18119 (thunderbird-bin).
Jun 28 12:09:09 soma Out of Memory: Killed process 8857 (thunderbird-bin).

Re: oom-killings, but I'm not out of memory!

[Alexander Nyberg]

> I'm running a 2.6.11 kernel.  I have 1 gig of RAM and 1 gig of swap.  Lately 
> when my RAM gets full, the oom-killer takes out either Mozilla or 
> Thunderbird (my two biggest memory hogs), even though my swap space is only 
> 20% full.  I still have ~800 MB of free swap space, so shouldn't the kernel 
> push Moz or T-bird into swap instead of oom-killing it?  At their maximum 
> memory-hogging capacity, neither Moz nor T-bird is ever using more than 200 MB.
>
> Jun 28 12:09:09 soma oom-killer: gfp_mask=0x80d2
> ...
> Jun 28 12:09:09 soma Free swap  = 781012kB
> Jun 28 12:09:09 soma Total swap = 987988kB
> Jun 28 12:09:09 soma Out of Memory: Killed process 30787 (thunderbird-bin).
> Jun 28 12:09:09 soma Out of Memory: Killed process 18112 (thunderbird-bin).
> Jun 28 12:09:09 soma Out of Memory: Killed process 18116 (thunderbird-bin).
> Jun 28 12:09:09 soma Out of Memory: Killed process 18117 (thunderbird-bin).
> Jun 28 12:09:09 soma Out of Memory: Killed process 18119 (thunderbird-bin).
> Jun 28 12:09:09 soma Out of Memory: Killed process 8857 (thunderbird-bin).
> 

You cut out the important part where it printed out memory usage
information at the time of the OOM, please post it

Re: oom-killings, but I'm not out of memory!

[Anthony DiSante]

Alexander Nyberg wrote:
>>I'm running a 2.6.11 kernel.  I have 1 gig of RAM and 1 gig of swap.  Lately 
>>when my RAM gets full, the oom-killer takes out either Mozilla or 
>>Thunderbird (my two biggest memory hogs), even though my swap space is only 
>>20% full.  I still have ~800 MB of free swap space, so shouldn't the kernel 
>>push Moz or T-bird into swap instead of oom-killing it?  At their maximum 
>>memory-hogging capacity, neither Moz nor T-bird is ever using more than 200 MB.
>>
>>Jun 28 12:09:09 soma oom-killer: gfp_mask=0x80d2
>>...
>>Jun 28 12:09:09 soma Free swap  = 781012kB
>>Jun 28 12:09:09 soma Total swap = 987988kB
>>Jun 28 12:09:09 soma Out of Memory: Killed process 30787 (thunderbird-bin).
>>Jun 28 12:09:09 soma Out of Memory: Killed process 18112 (thunderbird-bin).
>>Jun 28 12:09:09 soma Out of Memory: Killed process 18116 (thunderbird-bin).
>>Jun 28 12:09:09 soma Out of Memory: Killed process 18117 (thunderbird-bin).
>>Jun 28 12:09:09 soma Out of Memory: Killed process 18119 (thunderbird-bin).
>>Jun 28 12:09:09 soma Out of Memory: Killed process 8857 (thunderbird-bin).
>>
> 
> You cut out the important part where it printed out memory usage
> information at the time of the OOM, please post it
> 

Oops.  I left that out because it line-wrapped so bad, and I didn't realize 
it was important.  Here it is:

... oom-killer: gfp_mask=0x80d2
... DMA per-cpu:
... cpu 0 hot: low 2, high 6, batch 1
... cpu 0 cold: low 0, high 2, batch 1
... Normal per-cpu:
... cpu 0 hot: low 32, high 96, batch 16
... cpu 0 cold: low 0, high 32, batch 16
... HighMem per-cpu:
... cpu 0 hot: low 14, high 42, batch 7
... cpu 0 cold: low 0, high 14, batch 7
...
... Free pages:       12536kB (112kB HighMem)
... Active:240797 inactive:2399 dirty:0 writeback:0 unstable:0 free:3134 
slab:7144 mapped:240597 pagetables:1073
... DMA free:4096kB min:68kB low:84kB high:100kB active:8260kB inactive:0kB 
present:16384kB pages_scanned:9052 all_unreclaimable? yes
... lowmem_reserve[]: 0 880 1007
... Normal free:8328kB min:3756kB low:4692kB high:5632kB active:827084kB 
inactive:9468kB present:901120kB pages_scanned:23361 all_unreclaimable? no
... lowmem_reserve[]: 0 0 1023
... HighMem free:112kB min:128kB low:160kB high:192kB active:127844kB 
inactive:128kB present:131008kB pages_scanned:135459 all_unreclaimable? yes
... lowmem_reserve[]: 0 0 0
... DMA: 0*4kB 28*8kB 16*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 
1*2048kB 0*4096kB = 4096kB
... Normal: 98*4kB 16*8kB 216*16kB 18*32kB 1*64kB 1*128kB 0*256kB 1*512kB 
1*1024kB 1*2048kB 0*4096kB = 8328kB
... HighMem: 0*4kB 2*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 
0*1024kB 0*2048kB 0*4096kB = 112kB
... Swap cache: add 166973, delete 149202, find 1714386/1723885, race 0+0
... Free swap  = 781012kB
... Total swap = 987988kB
... Out of Memory: Killed process 30787 (thunderbird-bin).
... Out of Memory: Killed process 18112 (thunderbird-bin).
... Out of Memory: Killed process 18116 (thunderbird-bin).
... Out of Memory: Killed process 18117 (thunderbird-bin).
... Out of Memory: Killed process 18119 (thunderbird-bin).
... Out of Memory: Killed process 8857 (thunderbird-bin).

-Anthony DiSante
http://nodivisions.com/

Re: oom-killings, but I'm not out of memory!

[Alexander Nyberg]

> >>I'm running a 2.6.11 kernel.  I have 1 gig of RAM and 1 gig of swap.  Lately 
> >>when my RAM gets full, the oom-killer takes out either Mozilla or 
> >>Thunderbird (my two biggest memory hogs), even though my swap space is only 
> >>20% full.  I still have ~800 MB of free swap space, so shouldn't the kernel 
> >>push Moz or T-bird into swap instead of oom-killing it?  At their maximum 
> >>memory-hogging capacity, neither Moz nor T-bird is ever using more than 200 MB.
> >>
> > You cut out the important part where it printed out memory usage
> > information at the time of the OOM, please post it
> > 
> 
> Oops.  I left that out because it line-wrapped so bad, and I didn't realize 
> it was important.  Here it is:
> 
> ... oom-killer: gfp_mask=0x80d2
> ... DMA per-cpu:
> ... cpu 0 hot: low 2, high 6, batch 1
> ... cpu 0 cold: low 0, high 2, batch 1
> ... Normal per-cpu:
> ... cpu 0 hot: low 32, high 96, batch 16
> ... cpu 0 cold: low 0, high 32, batch 16
> ... HighMem per-cpu:
> ... cpu 0 hot: low 14, high 42, batch 7
> ... cpu 0 cold: low 0, high 14, batch 7
> ...
> ... Free pages:       12536kB (112kB HighMem)
> ... Active:240797 inactive:2399 dirty:0 writeback:0 unstable:0 free:3134 
> slab:7144 mapped:240597 pagetables:1073
> ... DMA free:4096kB min:68kB low:84kB high:100kB active:8260kB inactive:0kB 
> present:16384kB pages_scanned:9052 all_unreclaimable? yes
> ... lowmem_reserve[]: 0 880 1007
> ... Normal free:8328kB min:3756kB low:4692kB high:5632kB active:827084kB 
> inactive:9468kB present:901120kB pages_scanned:23361 all_unreclaimable? no
> ... lowmem_reserve[]: 0 0 1023
> ... HighMem free:112kB min:128kB low:160kB high:192kB active:127844kB 
> inactive:128kB present:131008kB pages_scanned:135459 all_unreclaimable? yes
> ... lowmem_reserve[]: 0 0 0
> ... DMA: 0*4kB 28*8kB 16*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 
> 1*2048kB 0*4096kB = 4096kB
> ... Normal: 98*4kB 16*8kB 216*16kB 18*32kB 1*64kB 1*128kB 0*256kB 1*512kB 
> 1*1024kB 1*2048kB 0*4096kB = 8328kB
> ... HighMem: 0*4kB 2*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 
> 0*1024kB 0*2048kB 0*4096kB = 112kB
> ... Swap cache: add 166973, delete 149202, find 1714386/1723885, race 0+0
> ... Free swap  = 781012kB
> ... Total swap = 987988kB
> ... Out of Memory: Killed process 30787 (thunderbird-bin).
> ... Out of Memory: Killed process 18112 (thunderbird-bin).
> ... Out of Memory: Killed process 18116 (thunderbird-bin).
> ... Out of Memory: Killed process 18117 (thunderbird-bin).
> ... Out of Memory: Killed process 18119 (thunderbird-bin).
> ... Out of Memory: Killed process 8857 (thunderbird-bin).

Yeah this indeed looks strange. gfp_mask == GFP_HIGHUSER | __GFP_ZERO

iirc Andrea fixing up some all_unreclaimable bug in 2.6.11 but this
looks like that for some reason it didn't go into the Normal zone which
has plenty of free pages...

Re: oom-killings, but I'm not out of memory!

[Marcelo Tosatti]

On Wed, Jun 29, 2005 at 02:57:15PM +0200, Alexander Nyberg wrote:
> > >>I'm running a 2.6.11 kernel.  I have 1 gig of RAM and 1 gig of swap.  Lately 
> > >>when my RAM gets full, the oom-killer takes out either Mozilla or 
> > >>Thunderbird (my two biggest memory hogs), even though my swap space is only 
> > >>20% full.  I still have ~800 MB of free swap space, so shouldn't the kernel 
> > >>push Moz or T-bird into swap instead of oom-killing it?  At their maximum 
> > >>memory-hogging capacity, neither Moz nor T-bird is ever using more than 200 MB.
> > >>
> > > You cut out the important part where it printed out memory usage
> > > information at the time of the OOM, please post it
> > > 
> > 
> > Oops.  I left that out because it line-wrapped so bad, and I didn't realize 
> > it was important.  Here it is:
> > 
> > ... oom-killer: gfp_mask=0x80d2
> > ... DMA per-cpu:
> > ... cpu 0 hot: low 2, high 6, batch 1
> > ... cpu 0 cold: low 0, high 2, batch 1
> > ... Normal per-cpu:
> > ... cpu 0 hot: low 32, high 96, batch 16
> > ... cpu 0 cold: low 0, high 32, batch 16
> > ... HighMem per-cpu:
> > ... cpu 0 hot: low 14, high 42, batch 7
> > ... cpu 0 cold: low 0, high 14, batch 7
> > ...
> > ... Free pages:       12536kB (112kB HighMem)
> > ... Active:240797 inactive:2399 dirty:0 writeback:0 unstable:0 free:3134 
> > slab:7144 mapped:240597 pagetables:1073
> > ... DMA free:4096kB min:68kB low:84kB high:100kB active:8260kB inactive:0kB 
> > present:16384kB pages_scanned:9052 all_unreclaimable? yes
> > ... lowmem_reserve[]: 0 880 1007
> > ... Normal free:8328kB min:3756kB low:4692kB high:5632kB active:827084kB 
> > inactive:9468kB present:901120kB pages_scanned:23361 all_unreclaimable? no
> > ... lowmem_reserve[]: 0 0 1023
> > ... HighMem free:112kB min:128kB low:160kB high:192kB active:127844kB 
> > inactive:128kB present:131008kB pages_scanned:135459 all_unreclaimable? yes
> > ... lowmem_reserve[]: 0 0 0
> > ... DMA: 0*4kB 28*8kB 16*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 
> > 1*2048kB 0*4096kB = 4096kB
> > ... Normal: 98*4kB 16*8kB 216*16kB 18*32kB 1*64kB 1*128kB 0*256kB 1*512kB 
> > 1*1024kB 1*2048kB 0*4096kB = 8328kB
> > ... HighMem: 0*4kB 2*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 
> > 0*1024kB 0*2048kB 0*4096kB = 112kB
> > ... Swap cache: add 166973, delete 149202, find 1714386/1723885, race 0+0
> > ... Free swap  = 781012kB
> > ... Total swap = 987988kB
> > ... Out of Memory: Killed process 30787 (thunderbird-bin).
> > ... Out of Memory: Killed process 18112 (thunderbird-bin).
> > ... Out of Memory: Killed process 18116 (thunderbird-bin).
> > ... Out of Memory: Killed process 18117 (thunderbird-bin).
> > ... Out of Memory: Killed process 18119 (thunderbird-bin).
> > ... Out of Memory: Killed process 8857 (thunderbird-bin).
> 
> Yeah this indeed looks strange. gfp_mask == GFP_HIGHUSER | __GFP_ZERO
> 
> iirc Andrea fixing up some all_unreclaimable bug in 2.6.11 but this
> looks like that for some reason it didn't go into the Normal zone which
> has plenty of free pages...

Anthony, are you certain that this was the only VM dump available? 

AFAICS the only possible explanation for a OOM kill manifestation under 
this conditions (free pages count in the normal zone is higher than 
its high watermark + lowmem reservation), is a higher order allocation.

Why the heck doesnt the OOM killer print the order of current allocation? 

Anyway, the current try_to_free_pages/alloc_pages interaction seem to 
continue susceptible to deliberate decisions, even after Nick changed
the algorithm to return success if the total count of reclaimed pages
is greater than SWAP_CLUSTER_MAX, instead of a single priority pass 
being greater than SWAP_CLUSTER_MAX.

It sounds to me that a more reliable indication of OOM is failure to 
free _any_ pages after a full priority decay.

Nick, Andrew? 

--- linux-2.6.11/mm/vmscan.c.orig       2005-07-03 11:02:15.000000000 -0300
+++ linux-2.6.11/mm/vmscan.c    2005-07-03 11:02:44.000000000 -0300
@@ -938,6 +938,8 @@
                if (sc.nr_scanned && priority < DEF_PRIORITY - 2)
                        blk_congestion_wait(WRITE, HZ/10);
        }
+       /* return failure only if zero pages have been reclaimed */
+       ret = !!total_reclaimed;
 out:
        for (i = 0; zones[i] != 0; i++)
                zones[i]->prev_priority = zones[i]->temp_priority;

Re: oom-killings, but I'm not out of memory!

[Roy Keene]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5005 bytes --]

I think I'm having the same issue.

I've 2 systems with 4GB of RAM and 2GB of swap that kill processes when 
they get a lot of disk I/O.  I've attached the full dmesg output which 
includes portions where it killed stuff despite having massive amounts of 
free memory.

 	Roy Keene
 	Planning Systems Inc.

On Sun, 3 Jul 2005, Marcelo Tosatti wrote:

>
> On Wed, Jun 29, 2005 at 02:57:15PM +0200, Alexander Nyberg wrote:
>>>>> I'm running a 2.6.11 kernel.  I have 1 gig of RAM and 1 gig of swap.  Lately
>>>>> when my RAM gets full, the oom-killer takes out either Mozilla or
>>>>> Thunderbird (my two biggest memory hogs), even though my swap space is only
>>>>> 20% full.  I still have ~800 MB of free swap space, so shouldn't the kernel
>>>>> push Moz or T-bird into swap instead of oom-killing it?  At their maximum
>>>>> memory-hogging capacity, neither Moz nor T-bird is ever using more than 200 MB.
>>>>>
>>>> You cut out the important part where it printed out memory usage
>>>> information at the time of the OOM, please post it
>>>>
>>>
>>> Oops.  I left that out because it line-wrapped so bad, and I didn't realize
>>> it was important.  Here it is:
>>>
>>> ... oom-killer: gfp_mask=0x80d2
>>> ... DMA per-cpu:
>>> ... cpu 0 hot: low 2, high 6, batch 1
>>> ... cpu 0 cold: low 0, high 2, batch 1
>>> ... Normal per-cpu:
>>> ... cpu 0 hot: low 32, high 96, batch 16
>>> ... cpu 0 cold: low 0, high 32, batch 16
>>> ... HighMem per-cpu:
>>> ... cpu 0 hot: low 14, high 42, batch 7
>>> ... cpu 0 cold: low 0, high 14, batch 7
>>> ...
>>> ... Free pages:       12536kB (112kB HighMem)
>>> ... Active:240797 inactive:2399 dirty:0 writeback:0 unstable:0 free:3134
>>> slab:7144 mapped:240597 pagetables:1073
>>> ... DMA free:4096kB min:68kB low:84kB high:100kB active:8260kB inactive:0kB
>>> present:16384kB pages_scanned:9052 all_unreclaimable? yes
>>> ... lowmem_reserve[]: 0 880 1007
>>> ... Normal free:8328kB min:3756kB low:4692kB high:5632kB active:827084kB
>>> inactive:9468kB present:901120kB pages_scanned:23361 all_unreclaimable? no
>>> ... lowmem_reserve[]: 0 0 1023
>>> ... HighMem free:112kB min:128kB low:160kB high:192kB active:127844kB
>>> inactive:128kB present:131008kB pages_scanned:135459 all_unreclaimable? yes
>>> ... lowmem_reserve[]: 0 0 0
>>> ... DMA: 0*4kB 28*8kB 16*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB
>>> 1*2048kB 0*4096kB = 4096kB
>>> ... Normal: 98*4kB 16*8kB 216*16kB 18*32kB 1*64kB 1*128kB 0*256kB 1*512kB
>>> 1*1024kB 1*2048kB 0*4096kB = 8328kB
>>> ... HighMem: 0*4kB 2*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB
>>> 0*1024kB 0*2048kB 0*4096kB = 112kB
>>> ... Swap cache: add 166973, delete 149202, find 1714386/1723885, race 0+0
>>> ... Free swap  = 781012kB
>>> ... Total swap = 987988kB
>>> ... Out of Memory: Killed process 30787 (thunderbird-bin).
>>> ... Out of Memory: Killed process 18112 (thunderbird-bin).
>>> ... Out of Memory: Killed process 18116 (thunderbird-bin).
>>> ... Out of Memory: Killed process 18117 (thunderbird-bin).
>>> ... Out of Memory: Killed process 18119 (thunderbird-bin).
>>> ... Out of Memory: Killed process 8857 (thunderbird-bin).
>>
>> Yeah this indeed looks strange. gfp_mask == GFP_HIGHUSER | __GFP_ZERO
>>
>> iirc Andrea fixing up some all_unreclaimable bug in 2.6.11 but this
>> looks like that for some reason it didn't go into the Normal zone which
>> has plenty of free pages...
>
> Anthony, are you certain that this was the only VM dump available?
>
> AFAICS the only possible explanation for a OOM kill manifestation under
> this conditions (free pages count in the normal zone is higher than
> its high watermark + lowmem reservation), is a higher order allocation.
>
> Why the heck doesnt the OOM killer print the order of current allocation?
>
> Anyway, the current try_to_free_pages/alloc_pages interaction seem to
> continue susceptible to deliberate decisions, even after Nick changed
> the algorithm to return success if the total count of reclaimed pages
> is greater than SWAP_CLUSTER_MAX, instead of a single priority pass
> being greater than SWAP_CLUSTER_MAX.
>
> It sounds to me that a more reliable indication of OOM is failure to
> free _any_ pages after a full priority decay.
>
> Nick, Andrew?
>
> --- linux-2.6.11/mm/vmscan.c.orig       2005-07-03 11:02:15.000000000 -0300
> +++ linux-2.6.11/mm/vmscan.c    2005-07-03 11:02:44.000000000 -0300
> @@ -938,6 +938,8 @@
>                if (sc.nr_scanned && priority < DEF_PRIORITY - 2)
>                        blk_congestion_wait(WRITE, HZ/10);
>        }
> +       /* return failure only if zero pages have been reclaimed */
> +       ret = !!total_reclaimed;
> out:
>        for (i = 0; zones[i] != 0; i++)
>                zones[i]->prev_priority = zones[i]->temp_priority;
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

[-- Attachment #2: Type: TEXT/PLAIN, Size: 15719 bytes --]

Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 185
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
Probing IDE interface ide0...
hda: LITE-ON DVDRW SOHW-1633S, ATAPI CD/DVD-ROM drive
Using cfq io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
Probing IDE interface ide2...
Probing IDE interface ide3...
Probing IDE interface ide4...
Probing IDE interface ide5...
hda: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
input: ImPS/2 Generic Wheel Mouse on isa0060/serio1
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP: routing cache hash table of 32768 buckets, 512Kbytes
TCP: Hash tables configured (established 262144 bind 43690)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S1 S4 S5)
ACPI wakeup devices: 
PXHA PXHB EPB0 EPC0 DBSA DBSB P0P1 USB1 USB2 USB3 EUSB UAR1 UAR2 PS2K PS2M 
Freeing unused kernel memory: 176k freed
SCSI subsystem initialized
megaraid cmm: 2.20.2.5 (Release Date: Fri Jan 21 00:01:03 EST 2005)
megaraid: 2.20.4.5 (Release Date: Thu Feb 03 12:27:22 EST 2005)
megaraid: probe new device 0x1000:0x0408:0x8086:0x3431: bus 4:slot 14:func 0
ACPI: PCI interrupt 0000:04:0e.0[A] -> GSI 86 (level, low) -> IRQ 225
megaraid: fw version:[514E] bios version:[H420]
scsi0 : LSI Logic MegaRAID driver
scsi[0]: scanning scsi channel 0 [Phy 0] for non-raid devices
  Vendor: SEAGATE   Model: DAT    DAT72-000  Rev: A060
  Type:   Sequential-Access                  ANSI SCSI revision: 03
scsi[0]: scanning scsi channel 1 [Phy 1] for non-raid devices
  Vendor: ESG-SHV   Model: SCA HSBP M27      Rev: 1.0B
  Type:   Processor                          ANSI SCSI revision: 02
scsi[0]: scanning scsi channel 2 [virtual] for logical drives
  Vendor: MegaRAID  Model: LD 0 RAID5  139G  Rev: 514E
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sda: 285155328 512-byte hdwr sectors (146000 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
 sda: sda1 sda2 sda3
Attached scsi disk sda at scsi0, channel 2, id 0, lun 0
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
st: Version 20040403, fixed bufsize 32768, s/g segs 256
Attached scsi tape st0 at scsi0, channel 0, id 6, lun 0
st0: try direct i/o: yes (alignment 512 B), max page reachable by HBA 4294967295
Attached scsi generic sg0 at scsi0, channel 0, id 6, lun 0,  type 1
Attached scsi generic sg1 at scsi0, channel 1, id 6, lun 0,  type 3
Attached scsi generic sg2 at scsi0, channel 2, id 0, lun 0,  type 0
inserting floppy driver for 2.6.9-11.ELsmp
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
Intel(R) PRO/1000 Network Driver - version 5.6.10.1-k2-NAPI
Copyright (c) 1999-2004 Intel Corporation.
ACPI: PCI interrupt 0000:08:04.0[A] -> GSI 52 (level, low) -> IRQ 209
divert: allocating divert_blk for eth0
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI interrupt 0000:08:04.1[B] -> GSI 53 (level, low) -> IRQ 217
divert: allocating divert_blk for eth1
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
hw_random hardware driver 1.0.0 loaded
ACPI: PCI interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 193
ehci_hcd 0000:00:1d.7: EHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: irq 193, pci mem f88aac00
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: USB 2.0 enabled, EHCI 1.00, driver 2004-May-10
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
USB Universal Host Controller Interface driver v2.2
ACPI: PCI interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169
uhci_hcd 0000:00:1d.0: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: irq 169, io base 0000bc00
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ACPI: PCI interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 177
uhci_hcd 0000:00:1d.1: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: irq 177, io base 0000c000
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
ACPI: PCI interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 185
uhci_hcd 0000:00:1d.2: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: irq 185, io base 0000c080
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ip_tables: (C) 2000-2002 Netfilter core team
usb 3-1: new low speed USB device using address 2
e1000: eth0: e1000_watchdog: NIC Link is Up 10 Mbps Half Duplex
e1000: eth1: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
ip_tables: (C) 2000-2002 Netfilter core team
hiddev96: USB HID v1.10 Device [American Power Conversion Smart-UPS 1000 XL FW:631.3.D USB FW:1] on usb-0000:00:1d.1-1
ACPI: Power Button (FF) [PWRF]
EXT3 FS on sda3, internal journal
device-mapper: 4.4.0-ioctl (2005-01-12) initialised: dm@uk.sistina.com
cdrom: open failed.
Adding 2096472k swap on /dev/sda2.  Priority:-1 extents:1
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
nbd: registered device at major 43
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec (nowayout= 1)
i2c /dev entries driver
NET: Registered protocol family 10
Disabled Privacy Extensions on device c03356c0(lo)
IPv6 over IPv4 tunneling driver
divert: not allocating divert_blk for non-ethernet device sit0
ip_tables: (C) 2000-2002 Netfilter core team
eth1: no IPv6 routers present
eth0: no IPv6 routers present
md: md0 stopped.
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:       14304kB (1664kB HighMem)
Active:7971 inactive:994335 dirty:327523 writeback:25721 unstable:0 free:3576 slab:29113 mapped:7996 pagetables:341
DMA free:12640kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:877 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:0kB min:928kB low:1856kB high:2784kB active:0kB inactive:739100kB present:901120kB pages_scanned:1556742 all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:31804kB inactive:3238320kB present:3801088kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 4*8kB 3*16kB 4*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12640kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
HighMem: 12*4kB 8*8kB 7*16kB 3*32kB 1*64kB 2*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
Swap cache: add 116, delete 17, find 0/0, race 0+0
Free swap:       2096008kB
1179648 pages of RAM
819168 pages of HIGHMEM
141773 reserved pages
1004523 pages shared
99 pages swap cached
Out of Memory: Killed process 3819 (named).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:       15264kB (2496kB HighMem)
Active:7690 inactive:994342 dirty:324822 writeback:28377 unstable:0 free:3816 slab:29090 mapped:7420 pagetables:324
DMA free:12640kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:878 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:128kB min:928kB low:1856kB high:2784kB active:0kB inactive:738920kB present:901120kB pages_scanned:1215654 all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:2496kB min:512kB low:1024kB high:1536kB active:30760kB inactive:3238320kB present:3801088kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 4*8kB 3*16kB 4*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12640kB
Normal: 6*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64kB
HighMem: 98*4kB 29*8kB 19*16kB 7*32kB 1*64kB 2*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 2496kB
Swap cache: add 116, delete 17, find 0/0, race 0+0
Free swap:       2096076kB
1179648 pages of RAM
819168 pages of HIGHMEM
141773 reserved pages
1003921 pages shared
99 pages swap cached
Out of Memory: Killed process 6211 (sendmail).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:       15968kB (3328kB HighMem)
Active:7489 inactive:994310 dirty:324570 writeback:28629 unstable:0 free:3992 slab:29006 mapped:7219 pagetables:313
DMA free:12640kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:878 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:0kB min:928kB low:1856kB high:2784kB active:0kB inactive:738920kB present:901120kB pages_scanned:2083917 all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:3328kB min:512kB low:1024kB high:1536kB active:29956kB inactive:3238320kB present:3801088kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 4*8kB 3*16kB 4*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12640kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
HighMem: 230*4kB 49*8kB 26*16kB 8*32kB 1*64kB 2*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 3328kB
Swap cache: add 116, delete 17, find 0/0, race 0+0
Free swap:       2096076kB
1179648 pages of RAM
819168 pages of HIGHMEM
141773 reserved pages
1003481 pages shared
99 pages swap cached
Out of Memory: Killed process 3956 (ntpd).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:       16608kB (3328kB HighMem)
Active:7530 inactive:994287 dirty:324242 writeback:28941 unstable:0 free:4152 slab:28823 mapped:7219 pagetables:313
DMA free:12640kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:879 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:704kB min:928kB low:1856kB high:2784kB active:8kB inactive:738980kB present:901120kB pages_scanned:7590 all_unreclaimable? no
protections[]: 0 0 0
HighMem free:3328kB min:512kB low:1024kB high:1536kB active:29960kB inactive:3238320kB present:3801088kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
DMA: 4*4kB 4*8kB 3*16kB 4*32kB 4*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12640kB
Normal: 74*4kB 23*8kB 14*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 704kB
HighMem: 230*4kB 49*8kB 26*16kB 8*32kB 1*64kB 2*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 3328kB
Swap cache: add 116, delete 17, find 0/0, race 0+0
Free swap:       2096076kB
1179648 pages of RAM
819168 pages of HIGHMEM
141773 reserved pages
1003435 pages shared
99 pages swap cached
Out of Memory: Killed process 4686 (ipfail).

Re: oom-killings, but I'm not out of memory!

[Marcelo Tosatti]

Hi Roy,

On Sun, Jul 03, 2005 at 09:44:37PM -0500, Roy Keene wrote:
> I think I'm having the same issue.
> 
> I've 2 systems with 4GB of RAM and 2GB of swap that kill processes when 
> they get a lot of disk I/O.  I've attached the full dmesg output which 
> includes portions where it killed stuff despite having massive amounts of 
> free memory.

What kernel version is that?

> oom-killer: gfp_mask=0xd0
> Mem-info:
> DMA per-cpu:
> cpu 0 hot: low 2, high 6, batch 1
> cpu 0 cold: low 0, high 2, batch 1
> cpu 1 hot: low 2, high 6, batch 1
> cpu 1 cold: low 0, high 2, batch 1
> cpu 2 hot: low 2, high 6, batch 1
> cpu 2 cold: low 0, high 2, batch 1
> cpu 3 hot: low 2, high 6, batch 1
> cpu 3 cold: low 0, high 2, batch 1
> Normal per-cpu:
> cpu 0 hot: low 32, high 96, batch 16
> cpu 0 cold: low 0, high 32, batch 16
> cpu 1 hot: low 32, high 96, batch 16
> cpu 1 cold: low 0, high 32, batch 16
> cpu 2 hot: low 32, high 96, batch 16
> cpu 2 cold: low 0, high 32, batch 16
> cpu 3 hot: low 32, high 96, batch 16
> cpu 3 cold: low 0, high 32, batch 16
> HighMem per-cpu:
> cpu 0 hot: low 32, high 96, batch 16
> cpu 0 cold: low 0, high 32, batch 16
> cpu 1 hot: low 32, high 96, batch 16
> cpu 1 cold: low 0, high 32, batch 16
> cpu 2 hot: low 32, high 96, batch 16
> cpu 2 cold: low 0, high 32, batch 16
> cpu 3 hot: low 32, high 96, batch 16
> cpu 3 cold: low 0, high 32, batch 16
> 
> Free pages:       14304kB (1664kB HighMem)
> Active:7971 inactive:994335 dirty:327523 writeback:25721 unstable:0 free:3576 slab:29113 mapped:7996 pagetables:341

There are about 100M of writeout data onflight - I suppose thats too much. 

Guess: can you switch to another IO scheduler than CFQ or reduce its queue size?

IIRC you can do that by reducing /sys/block/device/queue/nr_requests.

> DMA free:12640kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:877 all_unreclaimable? yes
> protections[]: 0 0 0
>
> Normal free:0kB min:928kB low:1856kB high:2784kB active:0kB inactive:739100kB present:901120kB pages_scanned:1556742 all_unreclaimable? yes
> protections[]: 0 0 0

You've got no reservations for the normal zone either. 

How does /proc/sys/vm/lowmem_reserve_ratio looks like?

Re: oom-killings, but I'm not out of memory!

[Roy Keene]

Howdy,

 	Roy Keene
 	Planning Systems Inc.

On Sun, 3 Jul 2005, Marcelo Tosatti wrote:

>
> Hi Roy,
>
> On Sun, Jul 03, 2005 at 09:44:37PM -0500, Roy Keene wrote:
>> I think I'm having the same issue.
>>
>> I've 2 systems with 4GB of RAM and 2GB of swap that kill processes when
>> they get a lot of disk I/O.  I've attached the full dmesg output which
>> includes portions where it killed stuff despite having massive amounts of
>> free memory.
>
> What kernel version is that?
>

Linux cog2 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux


>> oom-killer: gfp_mask=0xd0
>> Mem-info:
>> DMA per-cpu:
>> cpu 0 hot: low 2, high 6, batch 1
>> cpu 0 cold: low 0, high 2, batch 1
>> cpu 1 hot: low 2, high 6, batch 1
>> cpu 1 cold: low 0, high 2, batch 1
>> cpu 2 hot: low 2, high 6, batch 1
>> cpu 2 cold: low 0, high 2, batch 1
>> cpu 3 hot: low 2, high 6, batch 1
>> cpu 3 cold: low 0, high 2, batch 1
>> Normal per-cpu:
>> cpu 0 hot: low 32, high 96, batch 16
>> cpu 0 cold: low 0, high 32, batch 16
>> cpu 1 hot: low 32, high 96, batch 16
>> cpu 1 cold: low 0, high 32, batch 16
>> cpu 2 hot: low 32, high 96, batch 16
>> cpu 2 cold: low 0, high 32, batch 16
>> cpu 3 hot: low 32, high 96, batch 16
>> cpu 3 cold: low 0, high 32, batch 16
>> HighMem per-cpu:
>> cpu 0 hot: low 32, high 96, batch 16
>> cpu 0 cold: low 0, high 32, batch 16
>> cpu 1 hot: low 32, high 96, batch 16
>> cpu 1 cold: low 0, high 32, batch 16
>> cpu 2 hot: low 32, high 96, batch 16
>> cpu 2 cold: low 0, high 32, batch 16
>> cpu 3 hot: low 32, high 96, batch 16
>> cpu 3 cold: low 0, high 32, batch 16
>>
>> Free pages:       14304kB (1664kB HighMem)
>> Active:7971 inactive:994335 dirty:327523 writeback:25721 unstable:0 free:3576 slab:29113 mapped:7996 pagetables:341
>
> There are about 100M of writeout data onflight - I suppose thats too much.
>
> Guess: can you switch to another IO scheduler than CFQ or reduce its queue size?
>
> IIRC you can do that by reducing /sys/block/device/queue/nr_requests.
>

For all the devices I use this is set to 8192

I set "sda" to 512 now.

[root@cog2 ~]# cat /sys/block/{sda,nbd0,nbd1}/queue/nr_requests
8192
8192
8192


>> DMA free:12640kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:877 all_unreclaimable? yes
>> protections[]: 0 0 0
>>
>> Normal free:0kB min:928kB low:1856kB high:2784kB active:0kB inactive:739100kB present:901120kB pages_scanned:1556742 all_unreclaimable? yes
>> protections[]: 0 0 0
>
> You've got no reservations for the normal zone either.
>
> How does /proc/sys/vm/lowmem_reserve_ratio looks like?
>
>

It looks like it doesn't exist..
[root@cog2 ~]# ls /proc/sys/vm
block_dump              dirty_ratio                laptop_mode 
max_map_count    nr_pdflush_threads  page-cluster
dirty_background_ratio  dirty_writeback_centisecs  legacy_va_layout 
min_free_kbytes  overcommit_memory   swappiness
dirty_expire_centisecs  hugetlb_shm_group          lower_zone_protection 
nr_hugepages     overcommit_ratio    vfs_cache_pressure
[root@cog2 ~]#