2.6.32.5 regression: page allocation failure. order:1,

2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.

Now, suddenly the logs are full of "page allocation failure. order:1",
and the odd "page allocation failure. order:4" failures.

Wow.  WTF happened in 2.6.32 ???

Downgrading now.

Logs available if you want them:  http://rtr.ca/2.6.32_page_alloc_fail_logs.txt

--
Mark Lord

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mel Gorman]

On Tue, Jan 26, 2010 at 09:13:27PM -0500, Mark Lord wrote:
> I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
>
> Now, suddenly the logs are full of "page allocation failure. order:1",
> and the odd "page allocation failure. order:4" failures.
>
> Wow.  WTF happened in 2.6.32 ???
>

There was one bug related to MIGRATE_RESERVE that might be affecting
you. It reported as impacting swap-orientated workloads but it could
easily affect drivers that depend on high-order atomic allocations.
Unfortunately, the fix is not signed-off yet but I expect it to make its
way towards mainline when it is.

Here is the patch with a slightly-altered changelog. Can you test if it
makes a difference please?

==== CUT HERE ====
From: Hugh Dickins <hugh.dickins@tiscali.co.uk>
Subject: Fix 2.6.32 slowdown in heavy swapping

There is a problem with simply building kernels as part of a tmpfs loop
swapping tests, and it's only obvious on the PowerPC G5. The problem
is that those swapping builds run about 20% slower in 2.6.32 than
2.6.31 (and look as if they run increasingly slowly, though I'm not
certain of that); and surprisingly it bisected down to your commit
5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 page-allocator: split per-cpu list
into one-list-per-migrate-type

The problem was down to MIGRATE_RESERVE pages are being put on the
MIGRATE_MOVABLE list, then freed as MIGRATE_MOVABLE. While it is not clear
why this has such a severe impact, it may be down to how many short-lived
high-order allocations are taking place. On machines making large numbers of
short-lived-high-order allocations, they may be depending on MIGRATE_RESERVE
to allocate in a timely fashion. In the case where they are GFP_ATOMIC,
they may be depending on MIGRATE_RESERVE to just work.

The simplest, straight bugfix, patch is the one below: rely on
page_private instead of migratetype when freeing.

Unfortunately-not-signed-off-by: Hugh Dickins <hugh.dickins@tiscali.co.uk>
Acked-by: Mel Gorman <mel@csn.ul.ie>

---
 mm/page_alloc.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--- 2.6.33-rc1/mm/page_alloc.c	2009-12-18 11:42:54.000000000 +0000
+++ linux/mm/page_alloc.c	2009-12-20 19:10:50.000000000 +0000
@@ -555,8 +555,9 @@ static void free_pcppages_bulk(struct zo
 			page = list_entry(list->prev, struct page, lru);
 			/* must delete as __free_one_page list manipulates */
 			list_del(&page->lru);
-			__free_one_page(page, zone, 0, migratetype);
-			trace_mm_page_pcpu_drain(page, 0, migratetype);
+			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
+			__free_one_page(page, zone, 0, page_private(page));
+			trace_mm_page_pcpu_drain(page, 0, page_private(page));
 		} while (--count && --batch_free && !list_empty(list));
 	}
 	spin_unlock(&zone->lock);

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

Mel Gorman wrote:
> On Tue, Jan 26, 2010 at 09:13:27PM -0500, Mark Lord wrote:
>> I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
>>
>> Now, suddenly the logs are full of "page allocation failure. order:1",
>> and the odd "page allocation failure. order:4" failures.
>>
>> Wow.  WTF happened in 2.6.32 ???
>>
> 
> There was one bug related to MIGRATE_RESERVE that might be affecting
> you. It reported as impacting swap-orientated workloads but it could
> easily affect drivers that depend on high-order atomic allocations.
> Unfortunately, the fix is not signed-off yet but I expect it to make its
> way towards mainline when it is.
> 
> Here is the patch with a slightly-altered changelog. Can you test if it
> makes a difference please?
..

We don't like to reboot our 24/7 server very often,
and certainly not for debugging buggy kernels.

It's rock solid again with 2.6.31.12 on it now.

The defining characteristic of that machine, is that it has only 512MB
of physical RAM.  So perhaps I'll try booting a different machine here
with mem=512M and see how that behaves.  If the problem shows up on that,
then I'll try the patch.

Thanks.


> --- 2.6.33-rc1/mm/page_alloc.c	2009-12-18 11:42:54.000000000 +0000
> +++ linux/mm/page_alloc.c	2009-12-20 19:10:50.000000000 +0000
> @@ -555,8 +555,9 @@ static void free_pcppages_bulk(struct zo
>  			page = list_entry(list->prev, struct page, lru);
>  			/* must delete as __free_one_page list manipulates */
>  			list_del(&page->lru);
> -			__free_one_page(page, zone, 0, migratetype);
> -			trace_mm_page_pcpu_drain(page, 0, migratetype);
> +			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
> +			__free_one_page(page, zone, 0, page_private(page));
> +			trace_mm_page_pcpu_drain(page, 0, page_private(page));
>  		} while (--count && --batch_free && !list_empty(list));
>  	}
>  	spin_unlock(&zone->lock);

Re: 2.6.32.5 regression: page allocation failure. order:1,

[David Rientjes]

On Wed, 27 Jan 2010, Mark Lord wrote:

> > There was one bug related to MIGRATE_RESERVE that might be affecting
> > you. It reported as impacting swap-orientated workloads but it could
> > easily affect drivers that depend on high-order atomic allocations.
> > Unfortunately, the fix is not signed-off yet but I expect it to make its
> > way towards mainline when it is.
> > 
> > Here is the patch with a slightly-altered changelog. Can you test if it
> > makes a difference please?
> ..
> 
> We don't like to reboot our 24/7 server very often,
> and certainly not for debugging buggy kernels.
> 
> It's rock solid again with 2.6.31.12 on it now.
> 

Is there something specific about the workload that makes it easily 
reproducible?  Are you saying that 2.6.31.12 is "rock solid" because it 
has survived a certain workload that caused these page allocation failures 
with 2.6.32.5, or simply because it has a longer uptime and hasn't shown 
a problem?  It would be very helpful to describe the load so that we can 
attempt to reproduce it locally without a sacrifice to your server.

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

David Rientjes wrote:
> On Wed, 27 Jan 2010, Mark Lord wrote:
>> It's rock solid again with 2.6.31.12 on it now.
>>
> 
> Is there something specific about the workload that makes it easily 
> reproducible?  Are you saying that 2.6.31.12 is "rock solid" because it 
> has survived a certain workload that caused these page allocation failures 
> with 2.6.32.5, or simply because it has a longer uptime and hasn't shown 
> a problem?  It would be very helpful to describe the load so that we can 
> attempt to reproduce it locally without a sacrifice to your server.
..

Good questions, and I do "feel for you" here.  :)

That machine has a light workload.  Web server, email server,
dhcp for the small local network, name server, etc.

It sits at one end of a 6mb/1.5mb DSL connection, hardly busy at all.

But the logfile posted shows many of those "allocation failures"
within the first (only)36 hours of running 2.6.32.5:

  mirrordir (doing a backup to a USB drive)
  apcupsd  (UPS monitoring; hardly an intensive activity)
  apache2  (the web server receives rather light use, and no fancy php or anything)
  apache2  (again!)
  apache2  (and again)
  apache2  (another)
  apache2  (yet again)
  ...
  apache2  (and again again)
  vim      (hey, can't a guy edit his driver sources ??)
  cc1      (or compile them?)

So, not much happening, really.
It's a slow machine, a 600Mhz C7 ("VIA Samuel 2"), with only 512MB of RAM.

We've got 2.6.32.5 running on several other machines here with nary a glitch,
but all of those have 2-4GB of RAM.

Cheers

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

Mark Lord wrote:
> Mel Gorman wrote:
>> On Tue, Jan 26, 2010 at 09:13:27PM -0500, Mark Lord wrote:
>>> I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
>>>
>>> Now, suddenly the logs are full of "page allocation failure. order:1",
>>> and the odd "page allocation failure. order:4" failures.
>>>
>>> Wow.  WTF happened in 2.6.32 ???
>>>
>>
>> There was one bug related to MIGRATE_RESERVE that might be affecting
>> you. It reported as impacting swap-orientated workloads but it could
>> easily affect drivers that depend on high-order atomic allocations.
>> Unfortunately, the fix is not signed-off yet but I expect it to make its
>> way towards mainline when it is.
>>
>> Here is the patch with a slightly-altered changelog. Can you test if it
>> makes a difference please?
> ..
> 
> We don't like to reboot our 24/7 server very often,
> and certainly not for debugging buggy kernels.
> 
> It's rock solid again with 2.6.31.12 on it now.
> 
> The defining characteristic of that machine, is that it has only 512MB
> of physical RAM.  So perhaps I'll try booting a different machine here
> with mem=512M and see how that behaves.  If the problem shows up on that,
> then I'll try the patch.
..

Sod it.  2.6.32 is simply too broken for us here on 32-bit non-SMP.

Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook
locks up at boot time with several repeated messages like this:

   request_module: runaway loop modprobe binfmt_464c

Useless kernel on 32-bit.  I hope 2.6.33 ends up less buggy.

Cheers

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

Mark Lord wrote:
> Mark Lord wrote:
>> Mel Gorman wrote:
>>> On Tue, Jan 26, 2010 at 09:13:27PM -0500, Mark Lord wrote:
>>>> I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
>>>>
>>>> Now, suddenly the logs are full of "page allocation failure. order:1",
>>>> and the odd "page allocation failure. order:4" failures.
>>>>
>>>> Wow.  WTF happened in 2.6.32 ???
>>>>
>>>
>>> There was one bug related to MIGRATE_RESERVE that might be affecting
>>> you. It reported as impacting swap-orientated workloads but it could
>>> easily affect drivers that depend on high-order atomic allocations.
>>> Unfortunately, the fix is not signed-off yet but I expect it to make its
>>> way towards mainline when it is.
>>>
>>> Here is the patch with a slightly-altered changelog. Can you test if it
>>> makes a difference please?
>> ..
>>
>> We don't like to reboot our 24/7 server very often,
>> and certainly not for debugging buggy kernels.
>>
>> It's rock solid again with 2.6.31.12 on it now.
>>
>> The defining characteristic of that machine, is that it has only 512MB
>> of physical RAM.  So perhaps I'll try booting a different machine here
>> with mem=512M and see how that behaves.  If the problem shows up on that,
>> then I'll try the patch.
> ..
> 
> Sod it.  2.6.32 is simply too broken for us here on 32-bit non-SMP.
> 
> Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook
> locks up at boot time with several repeated messages like this:
> 
>   request_module: runaway loop modprobe binfmt_464c
> 
> Useless kernel on 32-bit.  I hope 2.6.33 ends up less buggy.
..

I rebuilt it (again!), this time as a pure UP (non-SMP) kernel,
and it still locks at boot, with or without the mem=512M parameter.

This is one really bad kernel release for 32-bit x86.

-ml

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mike Galbraith]

On Wed, 2010-01-27 at 23:34 -0500, Mark Lord wrote:
> Mark Lord wrote:
> > Mark Lord wrote:
> >> Mel Gorman wrote:
> >>> On Tue, Jan 26, 2010 at 09:13:27PM -0500, Mark Lord wrote:
> >>>> I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
> >>>>
> >>>> Now, suddenly the logs are full of "page allocation failure. order:1",
> >>>> and the odd "page allocation failure. order:4" failures.
> >>>>
> >>>> Wow.  WTF happened in 2.6.32 ???
> >>>>
> >>>
> >>> There was one bug related to MIGRATE_RESERVE that might be affecting
> >>> you. It reported as impacting swap-orientated workloads but it could
> >>> easily affect drivers that depend on high-order atomic allocations.
> >>> Unfortunately, the fix is not signed-off yet but I expect it to make its
> >>> way towards mainline when it is.
> >>>
> >>> Here is the patch with a slightly-altered changelog. Can you test if it
> >>> makes a difference please?
> >> ..
> >>
> >> We don't like to reboot our 24/7 server very often,
> >> and certainly not for debugging buggy kernels.
> >>
> >> It's rock solid again with 2.6.31.12 on it now.
> >>
> >> The defining characteristic of that machine, is that it has only 512MB
> >> of physical RAM.  So perhaps I'll try booting a different machine here
> >> with mem=512M and see how that behaves.  If the problem shows up on that,
> >> then I'll try the patch.
> > ..
> > 
> > Sod it.  2.6.32 is simply too broken for us here on 32-bit non-SMP.
> > 
> > Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook
> > locks up at boot time with several repeated messages like this:
> > 
> >   request_module: runaway loop modprobe binfmt_464c
> > 
> > Useless kernel on 32-bit.  I hope 2.6.33 ends up less buggy.
> ..
> 
> I rebuilt it (again!), this time as a pure UP (non-SMP) kernel,
> and it still locks at boot, with or without the mem=512M parameter.
> 
> This is one really bad kernel release for 32-bit x86.

FWFW, it doesn't seem to be anything generic.

My P4 box boots and runs fine nosmp+mem=512M.. no allocation errors so
far while beating it up.  It's not even a tiny bit fun to use with KDE
while doing so though. Doing a bunch of git and nfs activity convinces
reclaim that vital bits of my GUI are expendable.

	-Mike

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Rafael J. Wysocki]

On Thursday 28 January 2010, Mark Lord wrote:
> Mark Lord wrote:
> > Mark Lord wrote:
> >> Mel Gorman wrote:
> >>> On Tue, Jan 26, 2010 at 09:13:27PM -0500, Mark Lord wrote:
> >>>> I recently upgraded our 24/7 server from 2.6.31.5 to 2.6.32.5.
> >>>>
> >>>> Now, suddenly the logs are full of "page allocation failure. order:1",
> >>>> and the odd "page allocation failure. order:4" failures.
> >>>>
> >>>> Wow.  WTF happened in 2.6.32 ???
> >>>>
> >>>
> >>> There was one bug related to MIGRATE_RESERVE that might be affecting
> >>> you. It reported as impacting swap-orientated workloads but it could
> >>> easily affect drivers that depend on high-order atomic allocations.
> >>> Unfortunately, the fix is not signed-off yet but I expect it to make its
> >>> way towards mainline when it is.
> >>>
> >>> Here is the patch with a slightly-altered changelog. Can you test if it
> >>> makes a difference please?
> >> ..
> >>
> >> We don't like to reboot our 24/7 server very often,
> >> and certainly not for debugging buggy kernels.
> >>
> >> It's rock solid again with 2.6.31.12 on it now.
> >>
> >> The defining characteristic of that machine, is that it has only 512MB
> >> of physical RAM.  So perhaps I'll try booting a different machine here
> >> with mem=512M and see how that behaves.  If the problem shows up on that,
> >> then I'll try the patch.
> > ..
> > 
> > Sod it.  2.6.32 is simply too broken for us here on 32-bit non-SMP.
> > 
> > Attempting to boot a 32-bit kernel with "nosmp mem=512M" on my notebook
> > locks up at boot time with several repeated messages like this:
> > 
> >   request_module: runaway loop modprobe binfmt_464c
> > 
> > Useless kernel on 32-bit.  I hope 2.6.33 ends up less buggy.
> ..
> 
> I rebuilt it (again!), this time as a pure UP (non-SMP) kernel,
> and it still locks at boot, with or without the mem=512M parameter.
> 
> This is one really bad kernel release for 32-bit x86.

Have you attached a (2.6.31.x) dmesg output from the failing box somewhere
or have I just missed it?

Rafael

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

Rafael J. Wysocki wrote:
> Have you attached a (2.6.31.x) dmesg output from the failing box somewhere
> or have I just missed it?
..

The log from failing on 2.6.32.5 was linked to originally:

   http://rtr.ca/2.6.32_page_alloc_fail_logs.txt 

Here's the subsequent revert reboot back to 2.6.32.12:

   http://rtr.ca/2.6.31.12_ok_logs.txt

In the time available thus far, I haven't managed to get 2.6.32.5
to boot/run on anything else, using either a UP kernel, or "nosmp",
along with "mem=512M".

With a bit of luck, I'll have another similar VIA C7 box delivered here by tonight,
and should be able to boot/run that off a cloned disk image from the original system.

Then we can poke at things a bit more.
If the box works, it will be 100% at our disposal.

Note that it's far too slow for practical direct use of git-bisect,
taking hours to rebuild its own kernels.  It might be possible,
with a fair bit of effort, to arrange a quad-core box to do the
kernel compiles for it, though.

Cheers 

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

Mark Lord wrote:
> Rafael J. Wysocki wrote:
>> Have you attached a (2.6.31.x) dmesg output from the failing box 
>> somewhere
>> or have I just missed it?
> ..
> 
> The log from failing on 2.6.32.5 was linked to originally:
> 
>   http://rtr.ca/2.6.32_page_alloc_fail_logs.txt
> Here's the subsequent revert reboot back to 2.6.32.12:
> 
>   http://rtr.ca/2.6.31.12_ok_logs.txt
> 
> In the time available thus far, I haven't managed to get 2.6.32.5
> to boot/run on anything else, using either a UP kernel, or "nosmp",
> along with "mem=512M".
> 
> With a bit of luck, I'll have another similar VIA C7 box delivered here 
> by tonight,
> and should be able to boot/run that off a cloned disk image from the 
> original system.
> 
> Then we can poke at things a bit more.
> If the box works, it will be 100% at our disposal.
> 
> Note that it's far too slow for practical direct use of git-bisect,
> taking hours to rebuild its own kernels.  It might be possible,
> with a fair bit of effort, to arrange a quad-core box to do the
> kernel compiles for it, though.
..

Rather than wasting time trying to bisect a full major kernel revision,
I think instead I'll just focus on mm/page_alloc.c.

It appears the biggest changes were to get_page_from_freelist(),
so perhaps we'll just revert that one function and see how the
clone system behaves.

In the meanwhile, here's the diff -u of the two .config files:


--- 2.6.31/linux/.config	2010-01-26 21:38:36.000000000 -0500
+++ 2.6.32/linux/.config	2010-01-26 20:57:44.928354653 -0500
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
-# Linux kernel version: 2.6.31.12
-# Tue Jan 26 21:38:36 2010
+# Linux kernel version: 2.6.32.6
+# Tue Jan 26 20:57:44 2010
 #
 # CONFIG_64BIT is not set
 CONFIG_X86_32=y
@@ -16,7 +16,6 @@
 CONFIG_LOCKDEP_SUPPORT=y
 CONFIG_STACKTRACE_SUPPORT=y
 CONFIG_HAVE_LATENCYTOP_SUPPORT=y
-CONFIG_FAST_CMPXCHG_LOCAL=y
 CONFIG_MMU=y
 CONFIG_ZONE_DMA=y
 CONFIG_GENERIC_ISA_DMA=y
@@ -33,7 +32,8 @@
 CONFIG_ARCH_HAS_DEFAULT_IDLE=y
 CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
 CONFIG_HAVE_SETUP_PER_CPU_AREA=y
-CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y
+CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
+CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
 # CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
 CONFIG_ARCH_HIBERNATION_POSSIBLE=y
 CONFIG_ARCH_SUSPEND_POSSIBLE=y
@@ -76,11 +76,12 @@
 #
 # RCU Subsystem
 #
-CONFIG_CLASSIC_RCU=y
-# CONFIG_TREE_RCU is not set
-# CONFIG_PREEMPT_RCU is not set
+CONFIG_TREE_RCU=y
+# CONFIG_TREE_PREEMPT_RCU is not set
+# CONFIG_RCU_TRACE is not set
+CONFIG_RCU_FANOUT=8
+# CONFIG_RCU_FANOUT_EXACT is not set
 # CONFIG_TREE_RCU_TRACE is not set
-# CONFIG_PREEMPT_RCU_TRACE is not set
 CONFIG_IKCONFIG=y
 CONFIG_IKCONFIG_PROC=y
 CONFIG_LOG_BUF_SHIFT=15
@@ -123,21 +124,20 @@
 CONFIG_EVENTFD=y
 CONFIG_SHMEM=y
 CONFIG_AIO=y
-CONFIG_HAVE_PERF_COUNTERS=y
+CONFIG_HAVE_PERF_EVENTS=y
 
 #
-# Performance Counters
+# Kernel Performance Events And Counters
 #
+# CONFIG_PERF_EVENTS is not set
 # CONFIG_PERF_COUNTERS is not set
 CONFIG_VM_EVENT_COUNTERS=y
 CONFIG_PCI_QUIRKS=y
-CONFIG_STRIP_ASM_SYMS=y
 CONFIG_COMPAT_BRK=y
 CONFIG_SLAB=y
 # CONFIG_SLUB is not set
 # CONFIG_SLOB is not set
 # CONFIG_PROFILING is not set
-# CONFIG_MARKERS is not set
 CONFIG_HAVE_OPROFILE=y
 # CONFIG_KPROBES is not set
 CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
@@ -151,7 +151,7 @@
 #
 # GCOV-based kernel profiling
 #
-# CONFIG_SLOW_WORK is not set
+CONFIG_SLOW_WORK=y
 CONFIG_HAVE_GENERIC_DMA_COHERENT=y
 CONFIG_SLABINFO=y
 CONFIG_RT_MUTEXES=y
@@ -217,6 +217,7 @@
 # CONFIG_MVIAC7 is not set
 # CONFIG_MPSC is not set
 # CONFIG_MCORE2 is not set
+# CONFIG_MATOM is not set
 # CONFIG_GENERIC_CPU is not set
 # CONFIG_X86_GENERIC is not set
 CONFIG_X86_CPU=y
@@ -250,7 +251,6 @@
 # CONFIG_PREEMPT is not set
 # CONFIG_X86_UP_APIC is not set
 # CONFIG_X86_MCE is not set
-# CONFIG_X86_ANCIENT_MCE is not set
 CONFIG_VM86=y
 # CONFIG_TOSHIBA is not set
 # CONFIG_I8K is not set
@@ -283,6 +283,7 @@
 CONFIG_VIRT_TO_BUS=y
 CONFIG_HAVE_MLOCK=y
 CONFIG_HAVE_MLOCKED_PAGE_BIT=y
+# CONFIG_KSM is not set
 CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
 # CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
 # CONFIG_X86_RESERVE_LOW_64K is not set
@@ -313,6 +314,7 @@
 # CONFIG_PM_DEBUG is not set
 # CONFIG_SUSPEND is not set
 # CONFIG_HIBERNATION is not set
+# CONFIG_PM_RUNTIME is not set
 CONFIG_ACPI=y
 CONFIG_ACPI_PROCFS=y
 CONFIG_ACPI_PROCFS_POWER=y
@@ -324,6 +326,7 @@
 CONFIG_ACPI_FAN=m
 CONFIG_ACPI_DOCK=y
 CONFIG_ACPI_PROCESSOR=m
+# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
 CONFIG_ACPI_THERMAL=m
 # CONFIG_ACPI_CUSTOM_DSDT is not set
 CONFIG_ACPI_BLACKLIST_YEAR=0
@@ -332,6 +335,7 @@
 CONFIG_X86_PM_TIMER=y
 CONFIG_ACPI_CONTAINER=m
 # CONFIG_ACPI_SBS is not set
+# CONFIG_SFI is not set
 
 #
 # CPU Frequency scaling
@@ -626,6 +630,7 @@
 #
 # CONFIG_IP_DCCP_DEBUG is not set
 # CONFIG_IP_SCTP is not set
+# CONFIG_RDS is not set
 # CONFIG_TIPC is not set
 # CONFIG_ATM is not set
 CONFIG_STP=m
@@ -723,6 +728,7 @@
 # Generic Driver Options
 #
 CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
+# CONFIG_DEVTMPFS is not set
 CONFIG_STANDALONE=y
 CONFIG_PREVENT_FIRMWARE_BUILD=y
 CONFIG_FW_LOADER=y
@@ -816,6 +822,7 @@
 # CONFIG_SCSI_OSD_INITIATOR is not set
 CONFIG_ATA=y
 # CONFIG_ATA_NONSTANDARD is not set
+CONFIG_ATA_VERBOSE_ERROR=y
 CONFIG_ATA_ACPI=y
 # CONFIG_SATA_PMP is not set
 # CONFIG_SATA_AHCI is not set
@@ -839,6 +846,7 @@
 # CONFIG_PATA_ALI is not set
 # CONFIG_PATA_AMD is not set
 # CONFIG_PATA_ARTOP is not set
+# CONFIG_PATA_ATP867X is not set
 # CONFIG_PATA_ATIIXP is not set
 # CONFIG_PATA_CMD640_PCI is not set
 # CONFIG_PATA_CMD64X is not set
@@ -871,6 +879,7 @@
 # CONFIG_PATA_PDC_OLD is not set
 # CONFIG_PATA_QDI is not set
 # CONFIG_PATA_RADISYS is not set
+# CONFIG_PATA_RDC is not set
 # CONFIG_PATA_RZ1000 is not set
 # CONFIG_PATA_SC1200 is not set
 # CONFIG_PATA_SERVERWORKS is not set
@@ -973,6 +982,7 @@
 # CONFIG_SUNDANCE is not set
 # CONFIG_TLAN is not set
 # CONFIG_KS8842 is not set
+# CONFIG_KS8851_MLL is not set
 CONFIG_VIA_RHINE=y
 # CONFIG_VIA_RHINE_MMIO is not set
 # CONFIG_SC92031 is not set
@@ -1005,12 +1015,7 @@
 # CONFIG_JME is not set
 # CONFIG_NETDEV_10000 is not set
 # CONFIG_TR is not set
-
-#
-# Wireless LAN
-#
-# CONFIG_WLAN_PRE80211 is not set
-# CONFIG_WLAN_80211 is not set
+# CONFIG_WLAN is not set
 
 #
 # Enable WiMAX (Networking options) to see the WiMAX drivers
@@ -1063,6 +1068,7 @@
 # CONFIG_NETCONSOLE is not set
 # CONFIG_NETPOLL is not set
 # CONFIG_NET_POLL_CONTROLLER is not set
+# CONFIG_VMXNET3 is not set
 # CONFIG_ISDN is not set
 # CONFIG_PHONE is not set
 
@@ -1088,9 +1094,13 @@
 # Input Device Drivers
 #
 CONFIG_INPUT_KEYBOARD=y
+# CONFIG_KEYBOARD_ADP5588 is not set
 CONFIG_KEYBOARD_ATKBD=y
+# CONFIG_QT2160 is not set
 # CONFIG_KEYBOARD_LKKBD is not set
+# CONFIG_KEYBOARD_MAX7359 is not set
 # CONFIG_KEYBOARD_NEWTON is not set
+# CONFIG_KEYBOARD_OPENCORES is not set
 # CONFIG_KEYBOARD_STOWAWAY is not set
 # CONFIG_KEYBOARD_SUNKBD is not set
 # CONFIG_KEYBOARD_XTKBD is not set
@@ -1102,6 +1112,7 @@
 CONFIG_MOUSE_PS2_LIFEBOOK=y
 CONFIG_MOUSE_PS2_TRACKPOINT=y
 # CONFIG_MOUSE_PS2_ELANTECH is not set
+# CONFIG_MOUSE_PS2_SENTELIC is not set
 # CONFIG_MOUSE_PS2_TOUCHKIT is not set
 # CONFIG_MOUSE_SERIAL is not set
 # CONFIG_MOUSE_APPLETOUCH is not set
@@ -1125,6 +1136,7 @@
 CONFIG_INPUT_YEALINK=m
 # CONFIG_INPUT_CM109 is not set
 CONFIG_INPUT_UINPUT=m
+# CONFIG_INPUT_WINBOND_CIR is not set
 
 #
 # Hardware I/O ports
@@ -1224,6 +1236,7 @@
 CONFIG_DEVPORT=y
 CONFIG_I2C=m
 CONFIG_I2C_BOARDINFO=y
+CONFIG_I2C_COMPAT=y
 # CONFIG_I2C_CHARDEV is not set
 CONFIG_I2C_HELPER_AUTO=y
 CONFIG_I2C_ALGOBIT=m
@@ -1251,6 +1264,11 @@
 CONFIG_I2C_VIAPRO=m
 
 #
+# ACPI drivers
+#
+# CONFIG_I2C_SCMI is not set
+
+#
 # I2C system bus drivers (mostly embedded / system-on-chip)
 #
 # CONFIG_I2C_OCORES is not set
@@ -1282,9 +1300,6 @@
 # Miscellaneous I2C Chip support
 #
 # CONFIG_DS1682 is not set
-# CONFIG_SENSORS_PCF8574 is not set
-# CONFIG_PCF8575 is not set
-# CONFIG_SENSORS_PCA9539 is not set
 # CONFIG_SENSORS_TSL2550 is not set
 # CONFIG_I2C_DEBUG_CORE is not set
 # CONFIG_I2C_DEBUG_ALGO is not set
@@ -1320,6 +1335,7 @@
 # CONFIG_ALIM1535_WDT is not set
 # CONFIG_ALIM7101_WDT is not set
 # CONFIG_SC520_WDT is not set
+# CONFIG_SBC_FITPC2_WATCHDOG is not set
 # CONFIG_EUROTECH_WDT is not set
 # CONFIG_IB700_WDT is not set
 # CONFIG_IBMASR is not set
@@ -1377,6 +1393,7 @@
 # CONFIG_HTC_PASIC3 is not set
 # CONFIG_MFD_TMIO is not set
 # CONFIG_MFD_WM8400 is not set
+# CONFIG_MFD_WM831X is not set
 # CONFIG_MFD_WM8350_I2C is not set
 # CONFIG_MFD_PCF50633 is not set
 # CONFIG_AB3100_CORE is not set
@@ -1397,6 +1414,7 @@
 # CONFIG_AGP_SWORKS is not set
 CONFIG_AGP_VIA=m
 # CONFIG_AGP_EFFICEON is not set
+CONFIG_VGA_ARB=y
 CONFIG_DRM=m
 # CONFIG_DRM_TDFX is not set
 # CONFIG_DRM_R128 is not set
@@ -1480,6 +1498,7 @@
 # CONFIG_LOGO is not set
 CONFIG_SOUND=m
 CONFIG_SOUND_OSS_CORE=y
+CONFIG_SOUND_OSS_CORE_PRECLAIM=y
 CONFIG_SND=m
 CONFIG_SND_TIMER=m
 CONFIG_SND_PCM=m
@@ -1501,6 +1520,7 @@
 # CONFIG_SND_VERBOSE_PRINTK is not set
 # CONFIG_SND_DEBUG is not set
 CONFIG_SND_VMASTER=y
+CONFIG_SND_DMA_SGBUF=y
 CONFIG_SND_RAWMIDI_SEQ=m
 # CONFIG_SND_OPL3_LIB_SEQ is not set
 # CONFIG_SND_OPL4_LIB_SEQ is not set
@@ -1599,7 +1619,6 @@
 CONFIG_AC97_BUS=m
 CONFIG_HID_SUPPORT=y
 CONFIG_HID=m
-# CONFIG_HID_DEBUG is not set
 CONFIG_HIDRAW=y
 
 #
@@ -1623,6 +1642,7 @@
 CONFIG_HID_EZKEY=m
 CONFIG_HID_KYE=m
 CONFIG_HID_GYRATION=m
+CONFIG_HID_TWINHAN=m
 CONFIG_HID_KENSINGTON=m
 CONFIG_HID_LOGITECH=m
 # CONFIG_LOGITECH_FF is not set
@@ -1676,6 +1696,7 @@
 # CONFIG_USB_OXU210HP_HCD is not set
 # CONFIG_USB_ISP116X_HCD is not set
 # CONFIG_USB_ISP1760_HCD is not set
+# CONFIG_USB_ISP1362_HCD is not set
 CONFIG_USB_OHCI_HCD=m
 # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
 # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
@@ -1855,8 +1876,8 @@
 # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
 # CONFIG_EXT3_FS_XATTR is not set
 CONFIG_EXT4_FS=y
-# CONFIG_EXT4DEV_COMPAT is not set
 # CONFIG_EXT4_FS_XATTR is not set
+# CONFIG_EXT4_DEBUG is not set
 CONFIG_JBD=y
 CONFIG_JBD2=y
 # CONFIG_REISERFS_FS is not set
@@ -1865,6 +1886,7 @@
 # CONFIG_XFS_FS is not set
 # CONFIG_OCFS2_FS is not set
 # CONFIG_BTRFS_FS is not set
+# CONFIG_NILFS2_FS is not set
 CONFIG_FILE_LOCKING=y
 CONFIG_FSNOTIFY=y
 CONFIG_DNOTIFY=y
@@ -2003,6 +2025,7 @@
 # CONFIG_ENABLE_MUST_CHECK is not set
 CONFIG_FRAME_WARN=0
 CONFIG_MAGIC_SYSRQ=y
+CONFIG_STRIP_ASM_SYMS=y
 CONFIG_UNUSED_SYMBOLS=y
 # CONFIG_DEBUG_FS is not set
 # CONFIG_HEADERS_CHECK is not set
@@ -2039,6 +2062,7 @@
 # CONFIG_DEBUG_LIST is not set
 # CONFIG_DEBUG_SG is not set
 # CONFIG_DEBUG_NOTIFIERS is not set
+# CONFIG_DEBUG_CREDENTIALS is not set
 CONFIG_ARCH_WANT_FRAME_POINTERS=y
 # CONFIG_FRAME_POINTER is not set
 # CONFIG_BOOT_PRINTK_DELAY is not set
@@ -2046,6 +2070,7 @@
 # CONFIG_RCU_CPU_STALL_DETECTOR is not set
 # CONFIG_BACKTRACE_SELF_TEST is not set
 # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
+# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
 # CONFIG_FAULT_INJECTION is not set
 # CONFIG_LATENCYTOP is not set
 CONFIG_SYSCTL_SYSCALL_CHECK=y
@@ -2057,7 +2082,7 @@
 CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
 CONFIG_HAVE_DYNAMIC_FTRACE=y
 CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
-CONFIG_HAVE_FTRACE_SYSCALLS=y
+CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
 CONFIG_TRACING_SUPPORT=y
 # CONFIG_FTRACE is not set
 # CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
@@ -2118,7 +2143,7 @@
 CONFIG_CRYPTO_PCOMP=m
 CONFIG_CRYPTO_MANAGER=m
 CONFIG_CRYPTO_MANAGER2=m
-CONFIG_CRYPTO_GF128MUL=m
+CONFIG_CRYPTO_GF128MUL=y
 CONFIG_CRYPTO_NULL=m
 CONFIG_CRYPTO_WORKQUEUE=m
 CONFIG_CRYPTO_CRYPTD=m
@@ -2129,7 +2154,7 @@
 # Authenticated Encryption with Associated Data
 #
 CONFIG_CRYPTO_CCM=m
-CONFIG_CRYPTO_GCM=m
+# CONFIG_CRYPTO_GCM is not set
 CONFIG_CRYPTO_SEQIV=m
 
 #
@@ -2148,12 +2173,14 @@
 #
 CONFIG_CRYPTO_HMAC=m
 CONFIG_CRYPTO_XCBC=m
+CONFIG_CRYPTO_VMAC=m
 
 #
 # Digest
 #
 CONFIG_CRYPTO_CRC32C=m
 # CONFIG_CRYPTO_CRC32C_INTEL is not set
+# CONFIG_CRYPTO_GHASH is not set
 CONFIG_CRYPTO_MD4=m
 CONFIG_CRYPTO_MD5=m
 CONFIG_CRYPTO_MICHAEL_MIC=m
@@ -2208,7 +2235,6 @@
 # CONFIG_CRYPTO_DEV_GEODE is not set
 # CONFIG_CRYPTO_DEV_HIFN_795X is not set
 CONFIG_HAVE_KVM=y
-CONFIG_HAVE_KVM_IRQCHIP=y
 # CONFIG_VIRTUALIZATION is not set
 # CONFIG_BINARY_PRINTF is not set
 

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mel Gorman]

On Thu, Jan 28, 2010 at 09:17:17AM -0500, Mark Lord wrote:
> Mark Lord wrote:
>> Rafael J. Wysocki wrote:
>>> Have you attached a (2.6.31.x) dmesg output from the failing box  
>>> somewhere
>>> or have I just missed it?
>> ..
>>
>> The log from failing on 2.6.32.5 was linked to originally:
>>
>>   http://rtr.ca/2.6.32_page_alloc_fail_logs.txt
>> Here's the subsequent revert reboot back to 2.6.32.12:
>>
>>   http://rtr.ca/2.6.31.12_ok_logs.txt
>>
>> In the time available thus far, I haven't managed to get 2.6.32.5
>> to boot/run on anything else, using either a UP kernel, or "nosmp",
>> along with "mem=512M".
>>
>> With a bit of luck, I'll have another similar VIA C7 box delivered here 
>> by tonight,
>> and should be able to boot/run that off a cloned disk image from the  
>> original system.
>>
>> Then we can poke at things a bit more.
>> If the box works, it will be 100% at our disposal.
>>
>> Note that it's far too slow for practical direct use of git-bisect,
>> taking hours to rebuild its own kernels.  It might be possible,
>> with a fair bit of effort, to arrange a quad-core box to do the
>> kernel compiles for it, though.
> ..
>
> Rather than wasting time trying to bisect a full major kernel revision,
> I think instead I'll just focus on mm/page_alloc.c.
>

Well, it might not eve be necessary. In the patch I sent you, it pointed
the finger at commit 5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 being the
problem in that case. I believe your problem is a variation of the
slowdown-in-swapping problem except in your case it manifests as
GFP_ATOMIC allocations failing.

If the fix does not help you, then I'll take a fresh look at the other
commits with your particular problem in mind.

> It appears the biggest changes were to get_page_from_freelist(),
> so perhaps we'll just revert that one function and see how the
> clone system behaves.
>
> In the meanwhile, here's the diff -u of the two .config files:
>
>
> --- 2.6.31/linux/.config	2010-01-26 21:38:36.000000000 -0500
> +++ 2.6.32/linux/.config	2010-01-26 20:57:44.928354653 -0500
> @@ -1,7 +1,7 @@
> #
> # Automatically generated make config: don't edit
> -# Linux kernel version: 2.6.31.12
> -# Tue Jan 26 21:38:36 2010
> +# Linux kernel version: 2.6.32.6
> +# Tue Jan 26 20:57:44 2010
> #
> # CONFIG_64BIT is not set
> CONFIG_X86_32=y
> @@ -16,7 +16,6 @@
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> -CONFIG_FAST_CMPXCHG_LOCAL=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_GENERIC_ISA_DMA=y
> @@ -33,7 +32,8 @@
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> -CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y
> +CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> +CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> # CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> @@ -76,11 +76,12 @@
> #
> # RCU Subsystem
> #
> -CONFIG_CLASSIC_RCU=y
> -# CONFIG_TREE_RCU is not set
> -# CONFIG_PREEMPT_RCU is not set
> +CONFIG_TREE_RCU=y
> +# CONFIG_TREE_PREEMPT_RCU is not set
> +# CONFIG_RCU_TRACE is not set
> +CONFIG_RCU_FANOUT=8
> +# CONFIG_RCU_FANOUT_EXACT is not set
> # CONFIG_TREE_RCU_TRACE is not set
> -# CONFIG_PREEMPT_RCU_TRACE is not set
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=15
> @@ -123,21 +124,20 @@
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> -CONFIG_HAVE_PERF_COUNTERS=y
> +CONFIG_HAVE_PERF_EVENTS=y
>
> #
> -# Performance Counters
> +# Kernel Performance Events And Counters
> #
> +# CONFIG_PERF_EVENTS is not set
> # CONFIG_PERF_COUNTERS is not set
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> -CONFIG_STRIP_ASM_SYMS=y
> CONFIG_COMPAT_BRK=y
> CONFIG_SLAB=y
> # CONFIG_SLUB is not set
> # CONFIG_SLOB is not set
> # CONFIG_PROFILING is not set
> -# CONFIG_MARKERS is not set
> CONFIG_HAVE_OPROFILE=y
> # CONFIG_KPROBES is not set
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> @@ -151,7 +151,7 @@
> #
> # GCOV-based kernel profiling
> #
> -# CONFIG_SLOW_WORK is not set
> +CONFIG_SLOW_WORK=y
> CONFIG_HAVE_GENERIC_DMA_COHERENT=y
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> @@ -217,6 +217,7 @@
> # CONFIG_MVIAC7 is not set
> # CONFIG_MPSC is not set
> # CONFIG_MCORE2 is not set
> +# CONFIG_MATOM is not set
> # CONFIG_GENERIC_CPU is not set
> # CONFIG_X86_GENERIC is not set
> CONFIG_X86_CPU=y
> @@ -250,7 +251,6 @@
> # CONFIG_PREEMPT is not set
> # CONFIG_X86_UP_APIC is not set
> # CONFIG_X86_MCE is not set
> -# CONFIG_X86_ANCIENT_MCE is not set
> CONFIG_VM86=y
> # CONFIG_TOSHIBA is not set
> # CONFIG_I8K is not set
> @@ -283,6 +283,7 @@
> CONFIG_VIRT_TO_BUS=y
> CONFIG_HAVE_MLOCK=y
> CONFIG_HAVE_MLOCKED_PAGE_BIT=y
> +# CONFIG_KSM is not set
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> # CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
> # CONFIG_X86_RESERVE_LOW_64K is not set
> @@ -313,6 +314,7 @@
> # CONFIG_PM_DEBUG is not set
> # CONFIG_SUSPEND is not set
> # CONFIG_HIBERNATION is not set
> +# CONFIG_PM_RUNTIME is not set
> CONFIG_ACPI=y
> CONFIG_ACPI_PROCFS=y
> CONFIG_ACPI_PROCFS_POWER=y
> @@ -324,6 +326,7 @@
> CONFIG_ACPI_FAN=m
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=m
> +# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
> CONFIG_ACPI_THERMAL=m
> # CONFIG_ACPI_CUSTOM_DSDT is not set
> CONFIG_ACPI_BLACKLIST_YEAR=0
> @@ -332,6 +335,7 @@
> CONFIG_X86_PM_TIMER=y
> CONFIG_ACPI_CONTAINER=m
> # CONFIG_ACPI_SBS is not set
> +# CONFIG_SFI is not set
>
> #
> # CPU Frequency scaling
> @@ -626,6 +630,7 @@
> #
> # CONFIG_IP_DCCP_DEBUG is not set
> # CONFIG_IP_SCTP is not set
> +# CONFIG_RDS is not set
> # CONFIG_TIPC is not set
> # CONFIG_ATM is not set
> CONFIG_STP=m
> @@ -723,6 +728,7 @@
> # Generic Driver Options
> #
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> +# CONFIG_DEVTMPFS is not set
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> @@ -816,6 +822,7 @@
> # CONFIG_SCSI_OSD_INITIATOR is not set
> CONFIG_ATA=y
> # CONFIG_ATA_NONSTANDARD is not set
> +CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> # CONFIG_SATA_PMP is not set
> # CONFIG_SATA_AHCI is not set
> @@ -839,6 +846,7 @@
> # CONFIG_PATA_ALI is not set
> # CONFIG_PATA_AMD is not set
> # CONFIG_PATA_ARTOP is not set
> +# CONFIG_PATA_ATP867X is not set
> # CONFIG_PATA_ATIIXP is not set
> # CONFIG_PATA_CMD640_PCI is not set
> # CONFIG_PATA_CMD64X is not set
> @@ -871,6 +879,7 @@
> # CONFIG_PATA_PDC_OLD is not set
> # CONFIG_PATA_QDI is not set
> # CONFIG_PATA_RADISYS is not set
> +# CONFIG_PATA_RDC is not set
> # CONFIG_PATA_RZ1000 is not set
> # CONFIG_PATA_SC1200 is not set
> # CONFIG_PATA_SERVERWORKS is not set
> @@ -973,6 +982,7 @@
> # CONFIG_SUNDANCE is not set
> # CONFIG_TLAN is not set
> # CONFIG_KS8842 is not set
> +# CONFIG_KS8851_MLL is not set
> CONFIG_VIA_RHINE=y
> # CONFIG_VIA_RHINE_MMIO is not set
> # CONFIG_SC92031 is not set
> @@ -1005,12 +1015,7 @@
> # CONFIG_JME is not set
> # CONFIG_NETDEV_10000 is not set
> # CONFIG_TR is not set
> -
> -#
> -# Wireless LAN
> -#
> -# CONFIG_WLAN_PRE80211 is not set
> -# CONFIG_WLAN_80211 is not set
> +# CONFIG_WLAN is not set
>
> #
> # Enable WiMAX (Networking options) to see the WiMAX drivers
> @@ -1063,6 +1068,7 @@
> # CONFIG_NETCONSOLE is not set
> # CONFIG_NETPOLL is not set
> # CONFIG_NET_POLL_CONTROLLER is not set
> +# CONFIG_VMXNET3 is not set
> # CONFIG_ISDN is not set
> # CONFIG_PHONE is not set
>
> @@ -1088,9 +1094,13 @@
> # Input Device Drivers
> #
> CONFIG_INPUT_KEYBOARD=y
> +# CONFIG_KEYBOARD_ADP5588 is not set
> CONFIG_KEYBOARD_ATKBD=y
> +# CONFIG_QT2160 is not set
> # CONFIG_KEYBOARD_LKKBD is not set
> +# CONFIG_KEYBOARD_MAX7359 is not set
> # CONFIG_KEYBOARD_NEWTON is not set
> +# CONFIG_KEYBOARD_OPENCORES is not set
> # CONFIG_KEYBOARD_STOWAWAY is not set
> # CONFIG_KEYBOARD_SUNKBD is not set
> # CONFIG_KEYBOARD_XTKBD is not set
> @@ -1102,6 +1112,7 @@
> CONFIG_MOUSE_PS2_LIFEBOOK=y
> CONFIG_MOUSE_PS2_TRACKPOINT=y
> # CONFIG_MOUSE_PS2_ELANTECH is not set
> +# CONFIG_MOUSE_PS2_SENTELIC is not set
> # CONFIG_MOUSE_PS2_TOUCHKIT is not set
> # CONFIG_MOUSE_SERIAL is not set
> # CONFIG_MOUSE_APPLETOUCH is not set
> @@ -1125,6 +1136,7 @@
> CONFIG_INPUT_YEALINK=m
> # CONFIG_INPUT_CM109 is not set
> CONFIG_INPUT_UINPUT=m
> +# CONFIG_INPUT_WINBOND_CIR is not set
>
> #
> # Hardware I/O ports
> @@ -1224,6 +1236,7 @@
> CONFIG_DEVPORT=y
> CONFIG_I2C=m
> CONFIG_I2C_BOARDINFO=y
> +CONFIG_I2C_COMPAT=y
> # CONFIG_I2C_CHARDEV is not set
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=m
> @@ -1251,6 +1264,11 @@
> CONFIG_I2C_VIAPRO=m
>
> #
> +# ACPI drivers
> +#
> +# CONFIG_I2C_SCMI is not set
> +
> +#
> # I2C system bus drivers (mostly embedded / system-on-chip)
> #
> # CONFIG_I2C_OCORES is not set
> @@ -1282,9 +1300,6 @@
> # Miscellaneous I2C Chip support
> #
> # CONFIG_DS1682 is not set
> -# CONFIG_SENSORS_PCF8574 is not set
> -# CONFIG_PCF8575 is not set
> -# CONFIG_SENSORS_PCA9539 is not set
> # CONFIG_SENSORS_TSL2550 is not set
> # CONFIG_I2C_DEBUG_CORE is not set
> # CONFIG_I2C_DEBUG_ALGO is not set
> @@ -1320,6 +1335,7 @@
> # CONFIG_ALIM1535_WDT is not set
> # CONFIG_ALIM7101_WDT is not set
> # CONFIG_SC520_WDT is not set
> +# CONFIG_SBC_FITPC2_WATCHDOG is not set
> # CONFIG_EUROTECH_WDT is not set
> # CONFIG_IB700_WDT is not set
> # CONFIG_IBMASR is not set
> @@ -1377,6 +1393,7 @@
> # CONFIG_HTC_PASIC3 is not set
> # CONFIG_MFD_TMIO is not set
> # CONFIG_MFD_WM8400 is not set
> +# CONFIG_MFD_WM831X is not set
> # CONFIG_MFD_WM8350_I2C is not set
> # CONFIG_MFD_PCF50633 is not set
> # CONFIG_AB3100_CORE is not set
> @@ -1397,6 +1414,7 @@
> # CONFIG_AGP_SWORKS is not set
> CONFIG_AGP_VIA=m
> # CONFIG_AGP_EFFICEON is not set
> +CONFIG_VGA_ARB=y
> CONFIG_DRM=m
> # CONFIG_DRM_TDFX is not set
> # CONFIG_DRM_R128 is not set
> @@ -1480,6 +1498,7 @@
> # CONFIG_LOGO is not set
> CONFIG_SOUND=m
> CONFIG_SOUND_OSS_CORE=y
> +CONFIG_SOUND_OSS_CORE_PRECLAIM=y
> CONFIG_SND=m
> CONFIG_SND_TIMER=m
> CONFIG_SND_PCM=m
> @@ -1501,6 +1520,7 @@
> # CONFIG_SND_VERBOSE_PRINTK is not set
> # CONFIG_SND_DEBUG is not set
> CONFIG_SND_VMASTER=y
> +CONFIG_SND_DMA_SGBUF=y
> CONFIG_SND_RAWMIDI_SEQ=m
> # CONFIG_SND_OPL3_LIB_SEQ is not set
> # CONFIG_SND_OPL4_LIB_SEQ is not set
> @@ -1599,7 +1619,6 @@
> CONFIG_AC97_BUS=m
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=m
> -# CONFIG_HID_DEBUG is not set
> CONFIG_HIDRAW=y
>
> #
> @@ -1623,6 +1642,7 @@
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_GYRATION=m
> +CONFIG_HID_TWINHAN=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> # CONFIG_LOGITECH_FF is not set
> @@ -1676,6 +1696,7 @@
> # CONFIG_USB_OXU210HP_HCD is not set
> # CONFIG_USB_ISP116X_HCD is not set
> # CONFIG_USB_ISP1760_HCD is not set
> +# CONFIG_USB_ISP1362_HCD is not set
> CONFIG_USB_OHCI_HCD=m
> # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
> # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
> @@ -1855,8 +1876,8 @@
> # CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
> # CONFIG_EXT3_FS_XATTR is not set
> CONFIG_EXT4_FS=y
> -# CONFIG_EXT4DEV_COMPAT is not set
> # CONFIG_EXT4_FS_XATTR is not set
> +# CONFIG_EXT4_DEBUG is not set
> CONFIG_JBD=y
> CONFIG_JBD2=y
> # CONFIG_REISERFS_FS is not set
> @@ -1865,6 +1886,7 @@
> # CONFIG_XFS_FS is not set
> # CONFIG_OCFS2_FS is not set
> # CONFIG_BTRFS_FS is not set
> +# CONFIG_NILFS2_FS is not set
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> @@ -2003,6 +2025,7 @@
> # CONFIG_ENABLE_MUST_CHECK is not set
> CONFIG_FRAME_WARN=0
> CONFIG_MAGIC_SYSRQ=y
> +CONFIG_STRIP_ASM_SYMS=y
> CONFIG_UNUSED_SYMBOLS=y
> # CONFIG_DEBUG_FS is not set
> # CONFIG_HEADERS_CHECK is not set
> @@ -2039,6 +2062,7 @@
> # CONFIG_DEBUG_LIST is not set
> # CONFIG_DEBUG_SG is not set
> # CONFIG_DEBUG_NOTIFIERS is not set
> +# CONFIG_DEBUG_CREDENTIALS is not set
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> # CONFIG_FRAME_POINTER is not set
> # CONFIG_BOOT_PRINTK_DELAY is not set
> @@ -2046,6 +2070,7 @@
> # CONFIG_RCU_CPU_STALL_DETECTOR is not set
> # CONFIG_BACKTRACE_SELF_TEST is not set
> # CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
> +# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
> # CONFIG_FAULT_INJECTION is not set
> # CONFIG_LATENCYTOP is not set
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> @@ -2057,7 +2082,7 @@
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> -CONFIG_HAVE_FTRACE_SYSCALLS=y
> +CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_TRACING_SUPPORT=y
> # CONFIG_FTRACE is not set
> # CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
> @@ -2118,7 +2143,7 @@
> CONFIG_CRYPTO_PCOMP=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> -CONFIG_CRYPTO_GF128MUL=m
> +CONFIG_CRYPTO_GF128MUL=y
> CONFIG_CRYPTO_NULL=m
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_CRYPTD=m
> @@ -2129,7 +2154,7 @@
> # Authenticated Encryption with Associated Data
> #
> CONFIG_CRYPTO_CCM=m
> -CONFIG_CRYPTO_GCM=m
> +# CONFIG_CRYPTO_GCM is not set
> CONFIG_CRYPTO_SEQIV=m
>
> #
> @@ -2148,12 +2173,14 @@
> #
> CONFIG_CRYPTO_HMAC=m
> CONFIG_CRYPTO_XCBC=m
> +CONFIG_CRYPTO_VMAC=m
>
> #
> # Digest
> #
> CONFIG_CRYPTO_CRC32C=m
> # CONFIG_CRYPTO_CRC32C_INTEL is not set
> +# CONFIG_CRYPTO_GHASH is not set
> CONFIG_CRYPTO_MD4=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_MICHAEL_MIC=m
> @@ -2208,7 +2235,6 @@
> # CONFIG_CRYPTO_DEV_GEODE is not set
> # CONFIG_CRYPTO_DEV_HIFN_795X is not set
> CONFIG_HAVE_KVM=y
> -CONFIG_HAVE_KVM_IRQCHIP=y
> # CONFIG_VIRTUALIZATION is not set
> # CONFIG_BINARY_PRINTF is not set
>
>

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mark Lord]

Mel Gorman wrote:
> On Thu, Jan 28, 2010 at 09:17:17AM -0500, Mark Lord wrote:
..
>> Rather than wasting time trying to bisect a full major kernel revision,
>> I think instead I'll just focus on mm/page_alloc.c.
..
> Well, it might not eve be necessary. In the patch I sent you, it pointed
> the finger at commit 5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 being the
> problem in that case. I believe your problem is a variation of the
> slowdown-in-swapping problem except in your case it manifests as
> GFP_ATOMIC allocations failing.
> 
> If the fix does not help you, then I'll take a fresh look at the other
> commits with your particular problem in mind.
..

Last night, I installed 2.6.32.7, plus the patch you sent.
So far, no allocation faults.

I'll leave it running for another day or so, and then perhaps revert
the one patch to see which of the two things (new kernel, or patch)
is responsible for the difference.

The changelog for 2.6.32.7 included something to fix default rsize/wsize
values on NFS.  Dunno if this might have had an effect or not,
but when it was failing.. NFS (order 1) was the most frequent case.

-ml

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mel Gorman]

On Fri, Jan 29, 2010 at 08:56:20AM -0500, Mark Lord wrote:
> Mel Gorman wrote:
>> On Thu, Jan 28, 2010 at 09:17:17AM -0500, Mark Lord wrote:
> ..
>>> Rather than wasting time trying to bisect a full major kernel revision,
>>> I think instead I'll just focus on mm/page_alloc.c.
> ..
>> Well, it might not eve be necessary. In the patch I sent you, it pointed
>> the finger at commit 5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 being the
>> problem in that case. I believe your problem is a variation of the
>> slowdown-in-swapping problem except in your case it manifests as
>> GFP_ATOMIC allocations failing.
>>
>> If the fix does not help you, then I'll take a fresh look at the other
>> commits with your particular problem in mind.
> ..
>
> Last night, I installed 2.6.32.7, plus the patch you sent.
> So far, no allocation faults.
>

Nice one.

> I'll leave it running for another day or so, and then perhaps revert
> the one patch to see which of the two things (new kernel, or patch)
> is responsible for the difference.
>

Thanks, I'd appreciate it. While I'm reasonably confident the problem is
with MIGRATE_RESERVE not being free as intended and that the patch fixes
it, I'd like more proof.

> The changelog for 2.6.32.7 included something to fix default rsize/wsize
> values on NFS.  Dunno if this might have had an effect or not,
> but when it was failing.. NFS (order 1) was the most frequent case.
>

I consider it unlikely that a change in NFS is responsible. The network
MTU would remain the same and it's a bigger factor in the size of
allocations made. I think it's just a co-incidence that NFS reported the
most failures simply because it was responsible for the bulk of network
traffic.

Hugh, can I get a signed-off-by on that patch please? I can improve the
changelog if you like and send it to Andrew for merging if you like.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Hugh Dickins]

On Fri, 29 Jan 2010, Mel Gorman wrote:
> On Fri, Jan 29, 2010 at 08:56:20AM -0500, Mark Lord wrote:
> 
> > I'll leave it running for another day or so, and then perhaps revert
> > the one patch to see which of the two things (new kernel, or patch)
> > is responsible for the difference.
> >
> 
> Thanks, I'd appreciate it. While I'm reasonably confident the problem is
> with MIGRATE_RESERVE not being free as intended and that the patch fixes
> it, I'd like more proof.

You're more confident about that than I am!  It will be very satisfying
if my patch turns out to make the difference, but still surprising to me.

Thank you for taking the time on this, Mark: I too would appreciate it
if you could later determine whether it's new kernel or patch solving it.

> 
> Hugh, can I get a signed-off-by on that patch please? I can improve the
> changelog if you like and send it to Andrew for merging if you like.

I was adjusting the changelog and about to send direct to Linus Cc stable
in a few minutes, since I'm guessing there might be a 33-rc6 today, which
would be a pity to miss.

Whatever my reluctance to assume it's the fix to Mark's problem (which I'm
not mentioning in the changelog), we are both sure it's a valid bugfix.

Hugh

Re: 2.6.32.5 regression: page allocation failure. order:1,

[Mel Gorman]

On Fri, Jan 29, 2010 at 05:27:56PM +0000, Hugh Dickins wrote:
> On Fri, 29 Jan 2010, Mel Gorman wrote:
> > On Fri, Jan 29, 2010 at 08:56:20AM -0500, Mark Lord wrote:
> > 
> > > I'll leave it running for another day or so, and then perhaps revert
> > > the one patch to see which of the two things (new kernel, or patch)
> > > is responsible for the difference.
> > >
> > 
> > Thanks, I'd appreciate it. While I'm reasonably confident the problem is
> > with MIGRATE_RESERVE not being free as intended and that the patch fixes
> > it, I'd like more proof.
> 
> You're more confident about that than I am!  It will be very satisfying
> if my patch turns out to make the difference, but still surprising to me.
> 

My confidence/delusion is in part due to the reasons MIGRATE_RESERVE
exists in the first place. Specifically, certain wireless network drivers
were doing GFP_ATOMIC order-2 allocation a lot and failing miserably when
anti-fragmentation was first introduced. The problem came down to a property
of the buddy allocator implementation that kept min_free_kbytes worth of pages
free at the lower addresses of the zone and this is where order-2 allocations
were being made from and quickly freed meaning the area was generally available
for a lot of time.  MIGRATE_RESERVE was introduced to preserve that property
of the buddy allocator and the allocation failure problems went away.

Mark's problem feels very similar to the wireless network drivers problem.

Maybe I'm deluding myself.

> Thank you for taking the time on this, Mark: I too would appreciate it
> if you could later determine whether it's new kernel or patch solving it.
> 
> > 
> > Hugh, can I get a signed-off-by on that patch please? I can improve the
> > changelog if you like and send it to Andrew for merging if you like.
> 
> I was adjusting the changelog and about to send direct to Linus Cc stable
> in a few minutes, since I'm guessing there might be a 33-rc6 today, which
> would be a pity to miss.
> 

Great.

> Whatever my reluctance to assume it's the fix to Mark's problem (which I'm
> not mentioning in the changelog), we are both sure it's a valid bugfix.
> 

Indeed. Thanks.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

[PATCH] mm: fix migratetype bug which slowed swapping

[Hugh Dickins]

After memory pressure has forced it to dip into the reserves, 2.6.32's
5f8dcc21211a3d4e3a7a5ca366b469fb88117f61 "page-allocator: split per-cpu
list into one-list-per-migrate-type" has been returning MIGRATE_RESERVE
pages to the MIGRATE_MOVABLE free_list: in some sense depleting reserves.

Fix that in the most straightforward way (which, considering the overheads
of alternative approaches, is Mel's preference): the right migratetype is
already in page_private(page), but free_pcppages_bulk() wasn't using it.

How did this bug show up?  As a 20% slowdown in my tmpfs loop kbuild
swapping tests, on PowerMac G5 with SLUB allocator.  Bisecting to that
commit was easy, but explaining the magnitude of the slowdown not easy.

The same effect appears, but much less markedly, with SLAB, and even
less markedly on other machines (the PowerMac divides into fewer zones
than x86, I think that may be a factor).  We guess that lumpy reclaim
of short-lived high-order pages is implicated in some way, and probably
this bug has been tickling a poor decision somewhere in page reclaim.

But instrumentation hasn't told me much, I've run out of time and
imagination to determine exactly what's going on, and shouldn't hold up
the fix any longer: it's valid, and might even fix other misbehaviours.

Signed-off-by: Hugh Dickins <hugh.dickins@tiscali.co.uk>
Acked-by: Mel Gorman <mel@csn.ul.ie>
Cc: stable@kernel.org
---

 mm/page_alloc.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

--- 2.6.33-rc5/mm/page_alloc.c	2010-01-22 12:36:18.000000000 +0000
+++ linux/mm/page_alloc.c	2010-01-22 12:53:38.000000000 +0000
@@ -556,8 +556,9 @@ static void free_pcppages_bulk(struct zo
 			page = list_entry(list->prev, struct page, lru);
 			/* must delete as __free_one_page list manipulates */
 			list_del(&page->lru);
-			__free_one_page(page, zone, 0, migratetype);
-			trace_mm_page_pcpu_drain(page, 0, migratetype);
+			/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
+			__free_one_page(page, zone, 0, page_private(page));
+			trace_mm_page_pcpu_drain(page, 0, page_private(page));
 		} while (--count && --batch_free && !list_empty(list));
 	}
 	spin_unlock(&zone->lock);