[2.4] 0-order allocation failed

[2.4] 0-order allocation failed

[Michael Buesch]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

Hi all,

I'm running 2.4.28 bk snapshot of 2004.09.03
The machine has an uptime of 7 days, 23:46 now.

I was running several bittorrent clients inside of
a screen session. Suddenly they all died (including the
screen session).
dmesg sayed this:

__alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen

I already got this with kernel 2.4.27 vanilla after a
higher amount of uptime (I think it was over 10 days).
This was exactly the reason I updated to bk snapshot.

What can be the reason for this? Is it OOM? (I can't
really believe it is).
Is it a kernel memory leak?

With 2.4.26 I never got these errors. And I ran uptimes
up to 50 days.

-- 
Regards Michael Buesch  [ http://www.tuxsoft.de.vu ]


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.4] 0-order allocation failed

[Marcelo Tosatti]

On Thu, Oct 07, 2004 at 01:18:13PM +0200, Michael Buesch wrote:
> Hi all,
> 
> I'm running 2.4.28 bk snapshot of 2004.09.03
> The machine has an uptime of 7 days, 23:46 now.
> 
> I was running several bittorrent clients inside of
> a screen session. Suddenly they all died (including the
> screen session).
> dmesg sayed this:
> 
> __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
> __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> VM: killing process python
> __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> VM: killing process screen
> 
> I already got this with kernel 2.4.27 vanilla after a
> higher amount of uptime (I think it was over 10 days).
> This was exactly the reason I updated to bk snapshot.
> 
> What can be the reason for this? Is it OOM? (I can't
> really believe it is).

Can you check how much swap space is there available when
the OOM killer trigger? I bet this is the case.

If its not, we have a problem.

> Is it a kernel memory leak?
> 
> With 2.4.26 I never got these errors. And I ran uptimes
> up to 50 days.

Re: [2.4] 0-order allocation failed

[Marcelo Tosatti]

On Thu, Oct 07, 2004 at 07:17:30PM +0200, Michael Buesch wrote:
> Quoting Marcelo Tosatti <marcelo.tosatti@cyclades.com>:
> > On Thu, Oct 07, 2004 at 01:18:13PM +0200, Michael Buesch wrote:
> > > Hi all,
> > > 
> > > I'm running 2.4.28 bk snapshot of 2004.09.03
> > > The machine has an uptime of 7 days, 23:46 now.
> > > 
> > > I was running several bittorrent clients inside of
> > > a screen session. Suddenly they all died (including the
> > > screen session).
> > > dmesg sayed this:
> > > 
> > > __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
> > > __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> > > VM: killing process python
> > > __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> > > __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> > > VM: killing process screen
> > > 
> > > I already got this with kernel 2.4.27 vanilla after a
> > > higher amount of uptime (I think it was over 10 days).
> > > This was exactly the reason I updated to bk snapshot.
> > > 
> > > What can be the reason for this? Is it OOM? (I can't
> > > really believe it is).
> > 
> > Can you check how much swap space is there available when
> > the OOM killer trigger? I bet this is the case.
> 
> The machine doesn't have swap.

Well then you're probably facing true OOM.

Add some swap.

> > If its not, we have a problem.
> > 
> > > Is it a kernel memory leak?
> > > 
> > > With 2.4.26 I never got these errors. And I ran uptimes
> > > up to 50 days.

Re: [2.4] 0-order allocation failed

[Marcelo Tosatti]

On Thu, Oct 07, 2004 at 08:28:04PM +0200, Gabor Z. Papp wrote:
> * Marcelo Tosatti <marcelo.tosatti@cyclades.com>:
> 
> | > > Can you check how much swap space is there available when
> | > > the OOM killer trigger? I bet this is the case.
> | > 
> | > The machine doesn't have swap.
> | 
> | Well then you're probably facing true OOM.
> | 
> | Add some swap.
> 
> There is really no way to run 2.4 without swap?

Nope. Any kernel can't. The thing is the system overcommits 
memory (it allows applications to allocate more memory than the system 
is able to handle).

If there is no place to "save" that memory once you run out of it,
you're dead. Its a physical problem. :)

> I have the same problem with nfsroot and ramdisk based setups after
> 1-2 weeks uptime.

You can try to remove the following lines from mm/vmscan.c::try_to_free_pages_zone() 

        if (likely(current->pid != 1))
                break;
        if (!check_classzone_need_balance(classzone))
                break;

And disable CONFIG_OOM_KILLER. See if that makes a difference.

What will happen is that the kernel will try to free memory and 
never go into the OOM killer. If it can't free memory, the
system will hang forever at certain point.

There's not much to do about it really.

In 2.6 you can decrease swappiness so for it to free pagecache harder, 
but its the same game.

Re: [2.4] 0-order allocation failed

[Marcelo Tosatti]

On Thu, Oct 07, 2004 at 08:43:39PM +0200, Gabor Z. Papp wrote:
> * Marcelo Tosatti <marcelo.tosatti@cyclades.com>:
> 
> | > There is really no way to run 2.4 without swap?
> | 
> | Nope. Any kernel can't. The thing is the system overcommits 
> | memory (it allows applications to allocate more memory than the system 
> | is able to handle).
> 
> Okay, then whats the required minimum swap size that needed to avoid
> such crashes?
> 
> In the case when the system is in the ram, quite funny to allocate a
> swap file on the ramdisk anyway...

It depends on how much memory you have an idea of the workload
you have is going to use.

How much memory do you have? Try 128MB for a test.

Also investigate swap-over-nfs/swap-over-nbd.

Re: [2.4] 0-order allocation failed

[Michael Buesch]

[-- Attachment #1: Type: text/plain, Size: 1381 bytes --]

Quoting Marcelo Tosatti <marcelo.tosatti@cyclades.com>:
> On Thu, Oct 07, 2004 at 01:18:13PM +0200, Michael Buesch wrote:
> > Hi all,
> > 
> > I'm running 2.4.28 bk snapshot of 2004.09.03
> > The machine has an uptime of 7 days, 23:46 now.
> > 
> > I was running several bittorrent clients inside of
> > a screen session. Suddenly they all died (including the
> > screen session).
> > dmesg sayed this:
> > 
> > __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
> > __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> > VM: killing process python
> > __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> > __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> > VM: killing process screen
> > 
> > I already got this with kernel 2.4.27 vanilla after a
> > higher amount of uptime (I think it was over 10 days).
> > This was exactly the reason I updated to bk snapshot.
> > 
> > What can be the reason for this? Is it OOM? (I can't
> > really believe it is).
> 
> Can you check how much swap space is there available when
> the OOM killer trigger? I bet this is the case.

The machine doesn't have swap.

> If its not, we have a problem.
> 
> > Is it a kernel memory leak?
> > 
> > With 2.4.26 I never got these errors. And I ran uptimes
> > up to 50 days.
> 
> 

-- 
Regards Michael Buesch  [ http://www.tuxsoft.de.vu ]


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.4] 0-order allocation failed

[Gabor Z. Papp]

* Marcelo Tosatti <marcelo.tosatti@cyclades.com>:

| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > 
| > The machine doesn't have swap.
| 
| Well then you're probably facing true OOM.
| 
| Add some swap.

There is really no way to run 2.4 without swap?

I have the same problem with nfsroot and ramdisk based setups after
1-2 weeks uptime.

Re: [2.4] 0-order allocation failed

[Gabor Z. Papp]

* Marcelo Tosatti <marcelo.tosatti@cyclades.com>:

| > There is really no way to run 2.4 without swap?
| 
| Nope. Any kernel can't. The thing is the system overcommits 
| memory (it allows applications to allocate more memory than the system 
| is able to handle).

Okay, then whats the required minimum swap size that needed to avoid
such crashes?

In the case when the system is in the ram, quite funny to allocate a
swap file on the ramdisk anyway...

Re: [2.4] 0-order allocation failed

[Neil Horman]

Gabor Z. Papp wrote:
> * Marcelo Tosatti <marcelo.tosatti@cyclades.com>:
> 
> | > > Can you check how much swap space is there available when
> | > > the OOM killer trigger? I bet this is the case.
> | > 
> | > The machine doesn't have swap.
> | 
> | Well then you're probably facing true OOM.
> | 
> | Add some swap.
> 
> There is really no way to run 2.4 without swap?
> 
> I have the same problem with nfsroot and ramdisk based setups after
> 1-2 weeks uptime.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

sure, you can run a system without swap, you just run the risk of not 
being able to start new processes, or OOM kills if your running 
processees try to allocate too much memory.  You can turn the OOM killer 
off if you like, but then you run the risk of wedging the machine.

Neil

-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/

Re: [2.4] 0-order allocation failed

[Gabor Z. Papp]

* Marcelo Tosatti <marcelo.tosatti@cyclades.com>:

| Also investigate swap-over-nfs/swap-over-nbd.

I would like something *stable* solution, but I can't find
swap-over-nfs/swap-over-nbd in kernel. Thanks anyway.

Re: [2.4] 0-order allocation failed

[Marc-Christian Petersen]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:

Hi all,

> | > > Can you check how much swap space is there available when
> | > > the OOM killer trigger? I bet this is the case.
> | > The machine doesn't have swap.
> | Well then you're probably facing true OOM.
> | Add some swap.

> There is really no way to run 2.4 without swap?
> I have the same problem with nfsroot and ramdisk based setups after
> 1-2 weeks uptime.

stop whining about braindead 2.4 mainline vm. Apply the attached patch and be 
happy :p

Marcelo: Is there something wrong with my VM documentation update patches for 
2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM 
documentation even if almost all of the documentation is not longer valid"

;-)

ciao, Marc

[-- Attachment #2: vm-anon-lru-3.patch --]
[-- Type: text/x-diff, Size: 4605 bytes --]

diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/include/linux/swap.h x/include/linux/swap.h
--- x-ref/include/linux/swap.h	2003-10-10 08:08:29.000000000 +0200
+++ x/include/linux/swap.h	2003-10-17 21:18:14.000000000 +0200
@@ -115,7 +115,7 @@ extern void swap_setup(void);
 extern wait_queue_head_t kswapd_wait;
 extern int FASTCALL(try_to_free_pages_zone(zone_t *, unsigned int));
 extern int FASTCALL(try_to_free_pages(unsigned int));
-extern int vm_vfs_scan_ratio, vm_cache_scan_ratio, vm_lru_balance_ratio, vm_passes, vm_gfp_debug, vm_mapped_ratio;
+extern int vm_vfs_scan_ratio, vm_cache_scan_ratio, vm_lru_balance_ratio, vm_passes, vm_gfp_debug, vm_mapped_ratio, vm_anon_lru;
 
 /* linux/mm/page_io.c */
 extern void rw_swap_page(int, struct page *);
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/include/linux/sysctl.h x/include/linux/sysctl.h
--- x-ref/include/linux/sysctl.h	2003-10-17 21:18:12.000000000 +0200
+++ x/include/linux/sysctl.h	2003-10-17 21:18:46.000000000 +0200
@@ -154,9 +154,10 @@ enum
 	VM_PAGEBUF=17,		/* struct: Control pagebuf parameters */
 	VM_GFP_DEBUG=18,        /* debug GFP failures */
 	VM_CACHE_SCAN_RATIO=19, /* part of the inactive cache list to scan */
-	VM_MAPPED_RATIO=20,     /* amount of unfreeable pages that triggers swapout */
+	VM_MAPPED_RATIO=20,	/* amount of unfreeable pages that triggers swapout */
 	VM_LAPTOP_MODE=21,	/* kernel in laptop flush mode */
 	VM_BLOCK_DUMP=22,	/* dump fs activity to log */
+	VM_ANON_LRU=23,		/* immediatly insert anon pages in the vm page lru */
 };
 
 
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/kernel/sysctl.c x/kernel/sysctl.c
--- x-ref/kernel/sysctl.c	2003-10-17 21:18:12.000000000 +0200
+++ x/kernel/sysctl.c	2003-10-17 21:18:14.000000000 +0200
@@ -287,6 +287,8 @@ static ctl_table vm_table[] = {
 	 &vm_cache_scan_ratio, sizeof(int), 0644, NULL, &proc_dointvec},
 	{VM_MAPPED_RATIO, "vm_mapped_ratio", 
 	 &vm_mapped_ratio, sizeof(int), 0644, NULL, &proc_dointvec},
+	{VM_ANON_LRU, "vm_anon_lru", 
+	 &vm_anon_lru, sizeof(int), 0644, NULL, &proc_dointvec},
 	{VM_LRU_BALANCE_RATIO, "vm_lru_balance_ratio", 
 	 &vm_lru_balance_ratio, sizeof(int), 0644, NULL, &proc_dointvec},
 	{VM_PASSES, "vm_passes", 
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/mm/memory.c x/mm/memory.c
--- x-ref/mm/memory.c	2003-10-17 21:18:09.000000000 +0200
+++ x/mm/memory.c	2003-10-17 21:18:14.000000000 +0200
@@ -997,7 +997,8 @@ static int do_wp_page(struct mm_struct *
 		if (PageReserved(old_page))
 			++mm->rss;
 		break_cow(vma, new_page, address, page_table);
-		lru_cache_add(new_page);
+		if (vm_anon_lru)
+			lru_cache_add(new_page);
 
 		/* Free the old page.. */
 		new_page = old_page;
@@ -1228,7 +1229,8 @@ static int do_anonymous_page(struct mm_s
 		mm->rss++;
 		flush_page_to_ram(page);
 		entry = pte_mkwrite(pte_mkdirty(mk_pte(page, vma->vm_page_prot)));
-		lru_cache_add(page);
+		if (vm_anon_lru)
+			lru_cache_add(page);
 		mark_page_accessed(page);
 	}
 
@@ -1283,7 +1285,8 @@ static int do_no_page(struct mm_struct *
 		}
 		copy_user_highpage(page, new_page, address);
 		page_cache_release(new_page);
-		lru_cache_add(page);
+		if (vm_anon_lru)
+			lru_cache_add(page);
 		new_page = page;
 	}
 
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/mm/vmscan.c x/mm/vmscan.c
--- x-ref/mm/vmscan.c	2003-10-17 21:18:13.000000000 +0200
+++ x/mm/vmscan.c	2003-10-17 21:18:14.000000000 +0200
@@ -65,6 +65,27 @@ int vm_lru_balance_ratio = 2;
 int vm_vfs_scan_ratio = 6;
 
 /*
+ * "vm_anon_lru" select if to immdiatly insert anon pages in the
+ * lru. Immediatly means as soon as they're allocated during the
+ * page faults.
+ *
+ * If this is set to 0, they're inserted only after the first
+ * swapout.
+ *
+ * Having anon pages immediatly inserted in the lru allows the
+ * VM to know better when it's worthwhile to start swapping
+ * anonymous ram, it will start to swap earlier and it should
+ * swap smoother and faster, but it will decrease scalability
+ * on the >16-ways of an order of magnitude. Big SMP/NUMA
+ * definitely can't take an hit on a global spinlock at
+ * every anon page allocation. So this is off by default.
+ *
+ * Low ram machines that swaps all the time want to turn
+ * this on (i.e. set to 1).
+ */
+int vm_anon_lru = 0;
+
+/*
  * The swap-out function returns 1 if it successfully
  * scanned all the pages it was asked to (`count').
  * It returns zero if it couldn't do anything,

Re: [2.4] 0-order allocation failed

[Aleksandar Milivojevic]

Marcelo Tosatti wrote:
> On Thu, Oct 07, 2004 at 08:28:04PM +0200, Gabor Z. Papp wrote:
>>
>>There is really no way to run 2.4 without swap?
> 
> Nope. Any kernel can't. The thing is the system overcommits 
> memory (it allows applications to allocate more memory than the system 
> is able to handle).
> 
> If there is no place to "save" that memory once you run out of it,
> you're dead. Its a physical problem. :)

Hm, shouldn't there be command line option to instruct kernel not to 
overcommit memory?  For servers that don't really need disk, and 
wouldn't have much use for swap anyhow (like firewalls for example).  I 
do vaugly remember of such option in OSF/1 (Compaq Unix or whatever it 
is called nowdays) or Solaris.  Not sure which one.  Or maybe both.  If 
I remember correctly, there the default was not to overcommit, and (not 
recommended) option would allow kernel to overcommit.  Or I might be 
wrong about the default setting, but the option was there.  I don't 
remember if application that tried to allocate more memory that there 
was free on the system would be killed or if system call would simply fail.

-- 
Aleksandar Milivojevic <amilivojevic@pbl.ca>    Pollard Banknote Limited
Systems Administrator                           1499 Buffalo Place
Tel: (204) 474-2323 ext 276                     Winnipeg, MB  R3T 1L7

Re: [2.4] 0-order allocation failed

[Adam Heath]

On Thu, 7 Oct 2004, Gabor Z. Papp wrote:

> * Marcelo Tosatti <marcelo.tosatti@cyclades.com>:
>
> | Also investigate swap-over-nfs/swap-over-nbd.
>
> I would like something *stable* solution, but I can't find
> swap-over-nfs/swap-over-nbd in kernel. Thanks anyway.

google.

Re: [2.4] 0-order allocation failed

[Marcelo Tosatti]

On Thu, Oct 07, 2004 at 08:54:16PM +0200, Marc-Christian Petersen wrote:
> On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:
> 
> Hi all,
> 
> > | > > Can you check how much swap space is there available when
> > | > > the OOM killer trigger? I bet this is the case.
> > | > The machine doesn't have swap.
> > | Well then you're probably facing true OOM.
> > | Add some swap.
> 
> > There is really no way to run 2.4 without swap?
> > I have the same problem with nfsroot and ramdisk based setups after
> > 1-2 weeks uptime.
> 
> stop whining about braindead 2.4 mainline vm. Apply the attached patch and be 
> happy :p

As I told you in private, I can't see how badly this patch could affect performance.
But then, as you answered, with all anonymous pages added to LRU you see much better
behavior (tons less swapping) on several workloads. That must be due to 
refill_inactive()/shrink_cache() balancing.

The same patch also fixes kswapd excessive CPU consumption on huge
memory box.

Its easy enough to be applied because behaviour is unchanged by default
(you need to change a sysctl value for that).

I would like to understand why does it cause so much improved behaviour
though.

> Marcelo: Is there something wrong with my VM documentation update patches for 
> 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM 
> documentation even if almost all of the documentation is not longer valid"

As I said to you in private, please resend.

Thanks!

Re: [2.4] 0-order allocation failed

[Marcelo Tosatti]

On Thu, Oct 07, 2004 at 10:05:39PM -0300, Marcelo Tosatti wrote:
> On Thu, Oct 07, 2004 at 08:54:16PM +0200, Marc-Christian Petersen wrote:
> > On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:
> > 
> > Hi all,
> > 
> > > | > > Can you check how much swap space is there available when
> > > | > > the OOM killer trigger? I bet this is the case.
> > > | > The machine doesn't have swap.
> > > | Well then you're probably facing true OOM.
> > > | Add some swap.
> > 
> > > There is really no way to run 2.4 without swap?
> > > I have the same problem with nfsroot and ramdisk based setups after
> > > 1-2 weeks uptime.
> > 
> > stop whining about braindead 2.4 mainline vm. Apply the attached patch and be 
> > happy :p
> 
> As I told you in private, I can't see how badly this patch could affect performance.
> But then, as you answered, with all anonymous pages added to LRU you see much better
> behavior (tons less swapping) on several workloads. That must be due to 
> refill_inactive()/shrink_cache() balancing.

Ah, I dont think this will fix the OOM killer cases with no swap. They look 
like plain OOM condition to me.

Wish I'm wrong.

> The same patch also fixes kswapd excessive CPU consumption on huge
> memory box.
> 
> Its easy enough to be applied because behaviour is unchanged by default
> (you need to change a sysctl value for that).
> 
> I would like to understand why does it cause so much improved behaviour
> though.
> 
> > Marcelo: Is there something wrong with my VM documentation update patches for 
> > 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM 
> > documentation even if almost all of the documentation is not longer valid"
> 
> As I said to you in private, please resend.
> 

> Thanks!