public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
       [not found] <bug-11543-10286@http.bugzilla.kernel.org/>
@ 2008-09-12  0:02 ` Andrew Morton
  2008-09-12  2:54   ` j_kernel
  2008-09-12  9:13   ` Ingo Molnar
  0 siblings, 2 replies; 20+ messages in thread
From: Andrew Morton @ 2008-09-12  0:02 UTC (permalink / raw)
  To: linux-kernel, Ingo Molnar, Thomas Gleixner; +Cc: bugme-daemon, j_kernel


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Thu, 11 Sep 2008 16:46:29 -0700 (PDT)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=11543
> 
>            Summary: kernel panic: softlockup in tick_periodic() ???
>            Product: Platform Specific/Hardware
>            Version: 2.5
>      KernelVersion: 2.6.27-rc4-21704-gd25e26b
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: x86-64
>         AssignedTo: platform_x86_64@kernel-bugs.osdl.org
>         ReportedBy: j_kernel@hoblitt.com
> 

Is this a regression?  Was 2.6.26 OK, for example?

> [11532.103605] do_IRQ: 0.175 No irq handler for vector
> <Sep/11 12:13 pm>[11532.103613] do_IRQ: 2.175 No irq handler for vector
> <Sep/11 12:13 pm>[11532.103617] do_IRQ: 1.175 No irq handler for vector
> <Sep/11 12:14 pm>[11560.779989] do_IRQ: 0.179 No irq handler for vector
> <Sep/11 12:15 pm>[11622.181968] Kernel panic - not syncing: softlockup: hung
> tas<Sep/11 12:15 pm>
>                  <Sep/11 12:15 pm>[11622.181968] ------------[ cut here
> ]------------
> <Sep/11 12:15 pm>[11622.181968] WARNING: at kernel/mutex.c:351
> mutex_trylock+0x45/0xf6()
> <Sep/11 12:15 pm>[11622.181968] Modules linked in: w83627hf hwmon_vid autofs4
> smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs
> dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx
> scsi_wait_scan
> <Sep/11 12:15 pm>[11622.181968] Pid: 17192, comm: ppImage Not tainted
> 2.6.27-rc4-21704-gd25e26b #1
> <Sep/11 12:15 pm>[11622.181968] 
> <Sep/11 12:15 pm>[11622.181968] Call Trace:
> <Sep/11 12:15 pm>[11622.181968]  <IRQ>  [<ffffffff80235319>]
> warn_on_slowpath+0x51/0x77
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff802357ce>]
> release_console_sem+0x3e/0x1a1
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff805c6031>] mutex_trylock+0x45/0xf6
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8025efc3>] crash_kexec+0x17/0xef
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff803bb5b9>] bust_spinlocks+0x15/0x30
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff80235218>] panic+0x8f/0x13f
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff802357ce>]
> release_console_sem+0x3e/0x1a1
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff802357ce>]
> release_console_sem+0x3e/0x1a1
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff80272145>]
> softlockup_tick+0x19e/0x1ab
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8023dda4>]
> update_process_times+0x26/0x4b
> 
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8024f7a4>] tick_periodic+0x6e/0x79
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8024f7c7>]
> tick_handle_periodic+0x18/0x59
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8024f96a>]
> tick_do_broadcast+0x4d/0x86
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8024fa20>]
> tick_do_periodic_broadcast+0x23/0x31
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8024fa3c>]
> tick_handle_periodic_broadcast+0xe/0x42
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8020e9f6>]
> timer_event_interrupt+0x1a/0x21
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff80272591>]
> handle_IRQ_event+0x1e/0x4c
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff80273885>]
> handle_edge_irq+0xe8/0x12b
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8020e96f>] do_IRQ+0xf1/0x15e
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8020c3e1>] ret_from_intr+0x0/0xa
> <Sep/11 12:15 pm>[11622.181968]  <EOI>  [<ffffffff8021b79b>]
> native_flush_tlb_others+0x64/0xb3
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8021b7c5>]
> native_flush_tlb_others+0x8e/0xb3
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8021b7be>]
> native_flush_tlb_others+0x87/0xb3
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8021b8b2>] flush_tlb_page+0x5e/0x65
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff8022531b>]
> ptep_set_access_flags+0x1b/0x1f
> <Sep/11 12:15 pm>[11622.181968]  [<ffffffff80285193>] do_wp_page+0x48b/0x51e

argh, death by wordwrapping.

I can't work out who called panic(), nor why.

The panic code called the kexec code which called mutex_trylock() which
called spin_lock_mutex() which then stupidly went and blurted a load of
debug stuff because of in_interrupt().

Something like this:

--- a/include/linux/debug_locks.h~a
+++ a/include/linux/debug_locks.h
@@ -17,7 +17,7 @@ extern int debug_locks_off(void);
 ({									\
 	int __ret = 0;							\
 									\
-	if (unlikely(c)) {						\
+	if (!oops_in_progress && unlikely(c)) {				\
 		if (debug_locks_off() && !debug_locks_silent)		\
 			WARN_ON(1);					\
 		__ret = 1;						\
_

might prevent the debugging code from preventing us from finding bugs :(


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-12  0:02 ` Andrew Morton
@ 2008-09-12  2:54   ` j_kernel
  2008-09-12  2:57     ` Andrew Morton
  2008-09-12  9:13   ` Ingo Molnar
  1 sibling, 1 reply; 20+ messages in thread
From: j_kernel @ 2008-09-12  2:54 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel, Ingo Molnar, Thomas Gleixner, bugme-daemon

On Thu, Sep 11, 2008 at 05:02:58PM -0700, Andrew Morton wrote:
> Is this a regression?  Was 2.6.26 OK, for example?

It might be a regression. ;) The last build we were running on this
hardware was 2.6.24.2 and NMI watchdog support was not enabled.  We were
however experiencing random deadlocks, which I had been attributing to
problems with forcedeth.c (which causes the NIC to totally crap out
but not deadlock the machine) but I am now of the mind that there are
multiple problems with distinct failure modes.

> I can't work out who called panic(), nor why.

One more data point.  We booted this kernel on 14 machines this morning
and only one has had this panic thus far...

> The panic code called the kexec code which called mutex_trylock() which
> called spin_lock_mutex() which then stupidly went and blurted a load of
> debug stuff because of in_interrupt().
> 
> Something like this:
> 
> --- a/include/linux/debug_locks.h~a
> +++ a/include/linux/debug_locks.h
> @@ -17,7 +17,7 @@ extern int debug_locks_off(void);
>  ({									\
>  	int __ret = 0;							\
>  									\
> -	if (unlikely(c)) {						\
> +	if (!oops_in_progress && unlikely(c)) {				\
>  		if (debug_locks_off() && !debug_locks_silent)		\
>  			WARN_ON(1);					\
>  		__ret = 1;						\
> _
> 
> might prevent the debugging code from preventing us from finding bugs :(

Do you want me to give that patch a try or sit tight for a bit?

-J

--

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-12  2:54   ` j_kernel
@ 2008-09-12  2:57     ` Andrew Morton
  0 siblings, 0 replies; 20+ messages in thread
From: Andrew Morton @ 2008-09-12  2:57 UTC (permalink / raw)
  To: j_kernel; +Cc: linux-kernel, Ingo Molnar, Thomas Gleixner, bugme-daemon

On Thu, 11 Sep 2008 16:54:58 -1000 j_kernel@hoblitt.com wrote:

> > The panic code called the kexec code which called mutex_trylock() which
> > called spin_lock_mutex() which then stupidly went and blurted a load of
> > debug stuff because of in_interrupt().
> > 
> > Something like this:
> > 
> > --- a/include/linux/debug_locks.h~a
> > +++ a/include/linux/debug_locks.h
> > @@ -17,7 +17,7 @@ extern int debug_locks_off(void);
> >  ({									\
> >  	int __ret = 0;							\
> >  									\
> > -	if (unlikely(c)) {						\
> > +	if (!oops_in_progress && unlikely(c)) {				\
> >  		if (debug_locks_off() && !debug_locks_silent)		\
> >  			WARN_ON(1);					\
> >  		__ret = 1;						\
> > _
> > 
> > might prevent the debugging code from preventing us from finding bugs :(
> 
> Do you want me to give that patch a try or sit tight for a bit?

It's be good if you can try it please, see if we can get a cleaner
trace.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-12  0:02 ` Andrew Morton
  2008-09-12  2:54   ` j_kernel
@ 2008-09-12  9:13   ` Ingo Molnar
  2008-09-13  0:13     ` Joshua Hoblitt
  1 sibling, 1 reply; 20+ messages in thread
From: Ingo Molnar @ 2008-09-12  9:13 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel, Thomas Gleixner, bugme-daemon, j_kernel


* Andrew Morton <akpm@linux-foundation.org> wrote:

> I can't work out who called panic(), nor why.
> 
> The panic code called the kexec code which called mutex_trylock() 
> which called spin_lock_mutex() which then stupidly went and blurted a 
> load of debug stuff because of in_interrupt().

agreed - applied your fix in the form below to tip/master - thanks 
Andrew.

J, you might want to try tip/master, it includes all known fixes for 
this area and this debug improvement as well. You can pick it up via:

  http://people.redhat.com/mingo/tip.git/README

	Ingo

---------->
>From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001
From: Andrew Morton <akpm@linux-foundation.org>
Date: Thu, 11 Sep 2008 17:02:58 -0700
Subject: [PATCH] lock debug: sit tight when we are already in a panic

in:

  > http://bugzilla.kernel.org/show_bug.cgi?id=11543

The panic code called the kexec code which called mutex_trylock() which
called spin_lock_mutex() which then stupidly went and blurted a load of
debug stuff because of in_interrupt().

Keep the lock debug code from escallating an already crappy situation.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 include/linux/debug_locks.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h
index 4aaa4af..096476f 100644
--- a/include/linux/debug_locks.h
+++ b/include/linux/debug_locks.h
@@ -17,7 +17,7 @@ extern int debug_locks_off(void);
 ({									\
 	int __ret = 0;							\
 									\
-	if (unlikely(c)) {						\
+	if (!oops_in_progress && unlikely(c)) {				\
 		if (debug_locks_off() && !debug_locks_silent)		\
 			WARN_ON(1);					\
 		__ret = 1;						\

^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-12  9:13   ` Ingo Molnar
@ 2008-09-13  0:13     ` Joshua Hoblitt
  2008-09-15 21:06       ` Joshua Hoblitt
  0 siblings, 1 reply; 20+ messages in thread
From: Joshua Hoblitt @ 2008-09-13  0:13 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: Andrew Morton, linux-kernel, Thomas Gleixner, bugme-daemon,
	j_kernel

[-- Attachment #1: Type: text/plain, Size: 2240 bytes --]

I just rolled out -rc5 from netdev + Andrew's debug patch + the HPET
patch Thomas pointed me at.  I'll let it roast on these 14 machine is
production over the weekend to see if we get another panic.

I'm attaching the dmesg from this kernel.  We're still getting the NMI
watchdog warning and the rtc is [still] hosed (I think it was last
working around -rc3).

-J

--
On Fri, Sep 12, 2008 at 11:13:08AM +0200, Ingo Molnar wrote:
> 
> * Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > I can't work out who called panic(), nor why.
> > 
> > The panic code called the kexec code which called mutex_trylock() 
> > which called spin_lock_mutex() which then stupidly went and blurted a 
> > load of debug stuff because of in_interrupt().
> 
> agreed - applied your fix in the form below to tip/master - thanks 
> Andrew.
> 
> J, you might want to try tip/master, it includes all known fixes for 
> this area and this debug improvement as well. You can pick it up via:
> 
>   http://people.redhat.com/mingo/tip.git/README
> 
> 	Ingo
> 
> ---------->
> >From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001
> From: Andrew Morton <akpm@linux-foundation.org>
> Date: Thu, 11 Sep 2008 17:02:58 -0700
> Subject: [PATCH] lock debug: sit tight when we are already in a panic
> 
> in:
> 
>   > http://bugzilla.kernel.org/show_bug.cgi?id=11543
> 
> The panic code called the kexec code which called mutex_trylock() which
> called spin_lock_mutex() which then stupidly went and blurted a load of
> debug stuff because of in_interrupt().
> 
> Keep the lock debug code from escallating an already crappy situation.
> 
> Signed-off-by: Ingo Molnar <mingo@elte.hu>
> ---
>  include/linux/debug_locks.h |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h
> index 4aaa4af..096476f 100644
> --- a/include/linux/debug_locks.h
> +++ b/include/linux/debug_locks.h
> @@ -17,7 +17,7 @@ extern int debug_locks_off(void);
>  ({									\
>  	int __ret = 0;							\
>  									\
> -	if (unlikely(c)) {						\
> +	if (!oops_in_progress && unlikely(c)) {				\
>  		if (debug_locks_off() && !debug_locks_silent)		\
>  			WARN_ON(1);					\
>  		__ret = 1;						\

[-- Attachment #2: ipp018.dmesg --]
[-- Type: text/plain, Size: 43427 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Linux version 2.6.27-rc4-21704-gd25e26b (root@ipp000) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #1 SMP Tue Sep 2 16:05:45 HST 2008
[    0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
[    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000cff60000 (usable)
[    0.000000]  BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000230000000 (usable)
[    0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff
[    0.000000] init_memory_mapping
[    0.000000]  0000000000 - 00cfe00000 page 2M
[    0.000000]  00cfe00000 - 00cff60000 page 4k
[    0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000
[    0.000000] last_map_addr: cff60000 end: cff60000
[    0.000000] init_memory_mapping
[    0.000000]  0100000000 - 0230000000 page 2M
[    0.000000] kernel direct mapping tables up to 230000000 @ c000-16000
[    0.000000] last_map_addr: 230000000 end: 230000000
[    0.000000] RAMDISK: 37e0f000 - 37fef7dc
[    0.000000] DMI present.
[    0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD )
[    0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD  	 XSDT    6040000  LTP        0)
[    0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S      6040000 PTL_    F4240)
[    0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA      CK8  6040000 MSFT  3000000)
[    0.000000] ACPI: FACS CFF6AFC0, 0040
[    0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD    POWERNOW  6040000 AMD         1)
[    0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD    HAMMER    6040000 AMD         1)
[    0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
[    0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
[    0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD  HPETTBL   6040000  LTP        1)
[    0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD  	 APIC    6040000  LTP        0)
[    0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
[    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-d0000000
[    0.000000] SRAT: Node 0 PXM 0 100000000-130000000
[    0.000000] SRAT: Node 1 PXM 1 130000000-230000000
[    0.000000] NUMA: Allocated memnodemap from 11000 - 15680
[    0.000000] NUMA: Using 20 for the hash shift.
[    0.000000] Bootmem setup node 0 0000000000000000-0000000130000000
[    0.000000]   NODE_DATA [0000000000015680 - 000000000002067f]
[    0.000000]   bootmap [0000000000021000 -  0000000000046fff] pages 26
[    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
[    0.000000]   #2 [0000200000 - 0000f96890]    TEXT DATA BSS ==> [0000200000 - 0000f96890]
[    0.000000]   #3 [0037e0f000 - 0037fef7dc]          RAMDISK ==> [0037e0f000 - 0037fef7dc]
[    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
[    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE ==> [0000008000 - 000000c000]
[    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE ==> [000000c000 - 0000011000]
[    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP ==> [0000011000 - 0000015680]
[    0.000000] Bootmem setup node 1 0000000130000000-0000000230000000
[    0.000000]   NODE_DATA [0000000130000000 - 000000013000afff]
[    0.000000]   bootmap [000000013000b000 -  000000013002afff] pages 20
[    0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE
[    0.000000]   #2 [0000200000 - 0000f96890]    TEXT DATA BSS
[    0.000000]   #3 [0037e0f000 - 0037fef7dc]          RAMDISK
[    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved
[    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE
[    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE
[    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP
[    0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600
[    0.000000]  [ffffe20000000000-ffffe200071fffff] PMD -> [ffff880028200000-ffff88002e1fffff] on node 0
[    0.000000]  [ffffe20007200000-ffffe2000d1fffff] PMD -> [ffff880130200000-ffff8801361fffff] on node 1
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00230000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x0000009d
[    0.000000]     0: 0x00000100 -> 0x000cff60
[    0.000000]     0: 0x00100000 -> 0x00130000
[    0.000000]     1: 0x00130000 -> 0x00230000
[    0.000000] On node 0 totalpages: 1048317
[    0.000000]   DMA zone: 307 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 823232 pages, LIFO batch:31
[    0.000000]   Normal zone: 192000 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 1048576
[    0.000000]   Normal zone: 1024000 pages, LIFO batch:31
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x1008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Setting APIC routing to flat
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ce000
[    0.000000] PM: Registered nosave memory: 00000000000ce000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000cff60000 - 00000000cff6a000
[    0.000000] PM: Registered nosave memory: 00000000cff6a000 - 00000000cff80000
[    0.000000] PM: Registered nosave memory: 00000000cff80000 - 00000000d0000000
[    0.000000] PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000
[    0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
[    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
[    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000
[    0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fff80000
[    0.000000] PM: Registered nosave memory: 00000000fff80000 - 0000000100000000
[    0.000000] Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
[    0.000000] PERCPU: Allocating 51872 bytes of per cpu data
[    0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 2039539
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[    0.000000] Extended CMOS year: 2000
[    0.000000] TSC calibrated against PM_TIMER
[    0.000000] Detected 2800.009 MHz processor.
[    0.003333] spurious 8259A interrupt: IRQ7.
[    0.003333] Console: colour VGA+ 80x25
[    0.003333] console [tty0] enabled
[    0.003333] console [ttyS0] enabled
[    0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.003333] ... MAX_LOCKDEP_SUBCLASSES:    8
[    0.003333] ... MAX_LOCK_DEPTH:          48
[    0.003333] ... MAX_LOCKDEP_KEYS:        8191
[    0.003333] ... CLASSHASH_SIZE:           4096
[    0.003333] ... MAX_LOCKDEP_ENTRIES:     8192
[    0.003333] ... MAX_LOCKDEP_CHAINS:      16384
[    0.003333] ... CHAINHASH_SIZE:          8192
[    0.003333]  memory used by lock dependency info: 3839 kB
[    0.003333]  per task-struct memory footprint: 1920 bytes
[    0.003333] Checking aperture...
[    0.003333] No AGP bridge found
[    0.003333] Node 0: aperture @ 0 size 32 MB
[    0.003333] Your BIOS doesn't leave a aperture memory hole
[    0.003333] Please enable the IOMMU option in the BIOS setup
[    0.003333] This costs you 64 MB of RAM
[    0.003333] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.003333] PM: Registered nosave memory: 0000000020000000 - 0000000024000000
[    0.003333] Memory: 8108304k/9175040k available (3896k kernel code, 279268k reserved, 2243k data, 676k init)
[    0.003333] CPA: page pool initialized 1 of 1 pages preallocated
[    0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.003333] hpet clockevent registered
[    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.35 BogoMIPS (lpj=9333363)
[    0.006698] Security Framework initialized
[    0.010737] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.018937] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.025421] Mount-cache hash table entries: 256
[    0.026974] Initializing cgroup subsys ns
[    0.030010] Initializing cgroup subsys cpuacct
[    0.033335] Initializing cgroup subsys memory
[    0.036674] Initializing cgroup subsys devices
[    0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.043332] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.046666] CPU 0/0 -> Node 0
[    0.049999] tseg: 00cff80000
[    0.050000] CPU: Physical Processor ID: 0
[    0.053331] CPU: Processor Core ID: 0
[    0.056670] using C1E aware idle routine
[    0.060018] ACPI: Core revision 20080609
[    0.070530] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.108872] activating NMI Watchdog ... done.
[    0.109994] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.116660] Using local APIC timer interrupts.
[    0.123326] APIC timer calibration result 12500076
[    0.123328] Detected 12.500 MHz APIC timer.
[    0.126660] APIC timer registered as dummy, due to nmi_watchdog=1!
[    0.130154] lockdep: fixing up alternatives.
[    0.133347] Booting processor 1/1 ip 6000
[    0.003333] Initializing CPU#1
[    0.003333] Calibrating delay using timer specific routine.. 5602.36 BogoMIPS (lpj=9333385)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 1/1 -> Node 0
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
[    0.230372] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.238145] lockdep: fixing up alternatives.
[    0.239999] Booting processor 2/2 ip 6000
[    0.003333] Initializing CPU#2
[    0.003333] Calibrating delay using timer specific routine.. 5602.44 BogoMIPS (lpj=9333513)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 2/2 -> Node 1
[    0.003333] CPU: Physical Processor ID: 1
[    0.003333] CPU: Processor Core ID: 0
[    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 0x7010600070106
[    0.340530] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.348138] lockdep: fixing up alternatives.
[    0.349997] Booting processor 3/3 ip 6000
[    0.003333] Initializing CPU#3
[    0.003333] Calibrating delay using timer specific routine.. 5602.48 BogoMIPS (lpj=9333575)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 3/3 -> Node 1
[    0.003333] CPU: Physical Processor ID: 1
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106
[    0.447118] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.454734] Brought up 4 CPUs
[    0.456640] Total of 4 processors activated (22409.64 BogoMIPS).
[    0.460033] Testing NMI watchdog ... 
[    0.532432] WARNING: CPU#0: NMI appears to be stuck (0->0)!
[    0.533301] Please report this to bugzilla.kernel.org,
[    0.536635] and attach the output of the 'dmesg' command.
[    0.539968] 
[    0.543302] WARNING: CPU#1: NMI appears to be stuck (0->0)!
[    0.546635] Please report this to bugzilla.kernel.org,
[    0.549968] and attach the output of the 'dmesg' command.
[    0.553301] 
[    0.554980] WARNING: CPU#2: NMI appears to be stuck (0->0)!
[    0.556635] Please report this to bugzilla.kernel.org,
[    0.559968] and attach the output of the 'dmesg' command.
[    0.563301] 
[    0.566635] WARNING: CPU#3: NMI appears to be stuck (0->0)!
[    0.569968] Please report this to bugzilla.kernel.org,
[    0.573301] and attach the output of the 'dmesg' command.
[    0.577183] net_namespace: 968 bytes
[    0.580160] xor: automatically using best checksumming function: generic_sse
[    0.599969]    generic_sse:  8630.400 MB/sec
[    0.603302] xor: using function: generic_sse (8630.400 MB/sec)
[    0.606692] NET: Registered protocol family 16
[    0.610254] No dock devices found.
[    0.613503] node 0 link 2: io port [1000, 3fff]
[    0.613506] TOM: 00000000d0000000 aka 3328M
[    0.616636] node 0 link 2: mmio [d0000000, dfffffff]
[    0.616638] node 0 link 2: mmio [a0000, bffff]
[    0.616640] node 0 link 2: mmio [e0000000, e7ffffff]
[    0.616643] TOM2: 0000000230000000 aka 8960M
[    0.619969] bus: [00,ff] on node 0 link 2
[    0.619970] bus: 00 index 0 io port: [0, ffff]
[    0.619972] bus: 00 index 1 mmio: [d0000000, ffffffff]
[    0.619974] bus: 00 index 2 mmio: [a0000, bffff]
[    0.619975] bus: 00 index 3 mmio: [230000000, fcffffffff]
[    0.619989] ACPI: bus type pci registered
[    0.623448] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4
[    0.626636] PCI: MCFG area at e0000000 reserved in E820
[    0.630065] PCI: Using MMCONFIG at e0000000 - e04fffff
[    0.633302] PCI: Using configuration type 1 for base access
[    0.643980] ACPI: EC: Look up EC in DSDT
[    0.648409] ACPI: Interpreter enabled
[    0.649969] ACPI: (supports S0 S1 S3 S4 S5)
[    0.656635] ACPI: Using IOAPIC for interrupt routing
[    0.670034] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    0.673616] pci 0000:00:01.1: PME# supported from D3hot D3cold
[    0.676644] pci 0000:00:01.1: PME# disabled
[    0.680009] pci 0000:00:02.0: supports D1
[    0.680010] pci 0000:00:02.0: supports D2
[    0.680012] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.683303] pci 0000:00:02.0: PME# disabled
[    0.686672] pci 0000:00:02.1: supports D1
[    0.686673] pci 0000:00:02.1: supports D2
[    0.686675] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold
[    0.689969] pci 0000:00:02.1: PME# disabled
[    0.693535] pci 0000:00:08.0: supports D1
[    0.693537] pci 0000:00:08.0: supports D2
[    0.693538] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.696637] pci 0000:00:08.0: PME# disabled
[    0.700017] pci 0000:00:09.0: supports D1
[    0.700019] pci 0000:00:09.0: supports D2
[    0.700020] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.703304] pci 0000:00:09.0: PME# disabled
[    0.706664] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.709969] pci 0000:00:0a.0: PME# disabled
[    0.713331] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.716636] pci 0000:00:0d.0: PME# disabled
[    0.719997] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.723302] pci 0000:00:0f.0: PME# disabled
[    0.726878] pci 0000:01:04.0: supports D1
[    0.726879] pci 0000:01:04.0: supports D2
[    0.726904] pci 0000:00:06.0: transparent bridge
[    0.729969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff]
[    0.733303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff]
[    0.736636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff]
[    0.740034] pci 0000:02:00.0: supports D1
[    0.740064] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff]
[    0.743388] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.743462] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT]
[    0.743486] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT]
[    0.743513] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT]
[    0.743540] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT]
[    0.769795] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.777165] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.786790] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.797164] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.808626] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.816220] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.822660] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.830122] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.840505] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.851950] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.858235] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.867922] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 18 19 20 21 22 23)
[    0.877614] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.887303] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.898166] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.909241] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.915682] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.921892] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.928367] Linux Plug and Play Support v0.97 (c) Adam Belay
[    0.930030] pnp: PnP ACPI init
[    0.933313] ACPI: bus type pnp registered
[    0.942967] pnp: PnP ACPI: found 15 devices
[    0.943302] ACPI: ACPI bus type pnp unregistered
[    0.946899] SCSI subsystem initialized
[    0.950132] libata version 3.00 loaded.
[    0.953483] usbcore: registered new interface driver usbfs
[    0.956699] usbcore: registered new interface driver hub
[    0.960060] usbcore: registered new device driver usb
[    0.963671] PCI: Using ACPI for IRQ routing
[    0.980227] PCI-DMA: Disabling AGP.
[    0.984431] PCI-DMA: aperture base @ 20000000 size 65536 KB
[    0.986635] PCI-DMA: using GART IOMMU.
[    0.989971] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    0.993870] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
[    1.000341] hpet0: 3 32-bit timers, 25000000 Hz
[    1.004395] ACPI: RTC can wake from S4
[    1.006637] Clockevents: could not switch to one-shot mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional.
[    1.009524] Could not switch to high resolution mode on CPU 3
[    1.009528] Clockevents: could not switch to one-shot mode: lapic is not functional.
[    1.009531] Could not switch to high resolution mode on CPU 1
[    1.009534] Clockevents: could not switch to one-shot mode: lapic is not functional.
[    1.009538] Could not switch to high resolution mode on CPU 2
[    1.009969]  lapic is not functional.
[    1.056944] Could not switch to high resolution mode on CPU 0
[    1.076221] system 00:00: iomem range 0xffc00000-0xffffffff could not be reserved
[    1.084054] system 00:00: iomem range 0xfec00000-0xfec00fff could not be reserved
[    1.091883] system 00:00: iomem range 0xfee00000-0xfeefffff could not be reserved
[    1.099719] system 00:00: iomem range 0xfed00000-0xfed00fff has been reserved
[    1.107052] system 00:03: iomem range 0xd0000000-0xd0007fff has been reserved
[    1.114383] system 00:04: ioport range 0x1000-0x107f has been reserved
[    1.121101] system 00:04: ioport range 0x1080-0x10ff has been reserved
[    1.127817] system 00:04: ioport range 0x1400-0x147f has been reserved
[    1.134537] system 00:04: ioport range 0x1480-0x14ff has been reserved
[    1.141258] system 00:04: ioport range 0x1800-0x187f has been reserved
[    1.147976] system 00:04: ioport range 0x1880-0x18ff has been reserved
[    1.154698] system 00:04: ioport range 0x2440-0x247f has been reserved
[    1.161420] system 00:04: ioport range 0x2400-0x243f has been reserved
[    1.168151] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
[    1.174695] system 00:07: ioport range 0xc05-0xc06 has been reserved
[    1.182277] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01
[    1.188388] pci 0000:00:06.0:   IO window: 0x3000-0x3fff
[    1.193889] pci 0000:00:06.0:   MEM window: 0xd0100000-0xd01fffff
[    1.200176] pci 0000:00:06.0:   PREFETCH window: 0x000000d8000000-0x000000dfffffff
[    1.208089] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02
[    1.214205] pci 0000:00:0a.0:   IO window: disabled
[    1.219275] pci 0000:00:0a.0:   MEM window: 0xd0200000-0xd02fffff
[    1.225558] pci 0000:00:0a.0:   PREFETCH window: 0x000000d1000000-0x000000d10fffff
[    1.233474] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03
[    1.239598] pci 0000:00:0d.0:   IO window: disabled
[    1.244666] pci 0000:00:0d.0:   MEM window: disabled
[    1.249825] pci 0000:00:0d.0:   PREFETCH window: disabled
[    1.255416] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04
[    1.261525] pci 0000:00:0f.0:   IO window: disabled
[    1.266594] pci 0000:00:0f.0:   MEM window: disabled
[    1.271751] pci 0000:00:0f.0:   PREFETCH window: disabled
[    1.277347] pci 0000:00:06.0: setting latency timer to 64
[    1.277352] pci 0000:00:0a.0: setting latency timer to 64
[    1.277357] pci 0000:00:0d.0: setting latency timer to 64
[    1.277363] pci 0000:00:0f.0: setting latency timer to 64
[    1.277365] bus: 00 index 0 io port: [0, ffff]
[    1.282004] bus: 00 index 1 mmio: [0, ffffffffffffffff]
[    1.292287] bus: 01 index 0 io port: [3000, 3fff]
[    1.297187] bus: 01 index 1 mmio: [d0100000, d01fffff]
[    1.302519] bus: 01 index 2 mmio: [d8000000, dfffffff]
[    1.307849] bus: 01 index 3 io port: [0, ffff]
[    1.312483] bus: 01 index 4 mmio: [0, ffffffffffffffff]
[    1.317900] bus: 02 index 0 mmio: [0, 0]
[    1.322017] bus: 02 index 1 mmio: [d0200000, d02fffff]
[    1.327349] bus: 02 index 2 mmio: [d1000000, d10fffff]
[    1.332678] bus: 02 index 3 mmio: [0, 0]
[    1.336793] bus: 03 index 0 mmio: [0, 0]
[    1.340910] bus: 03 index 1 mmio: [0, 0]
[    1.345025] bus: 03 index 2 mmio: [0, 0]
[    1.349142] bus: 03 index 3 mmio: [0, 0]
[    1.353256] bus: 04 index 0 mmio: [0, 0]
[    1.357374] bus: 04 index 1 mmio: [0, 0]
[    1.361490] bus: 04 index 2 mmio: [0, 0]
[    1.365609] bus: 04 index 3 mmio: [0, 0]
[    1.369739] NET: Registered protocol family 2
[    1.413137] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    1.423160] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    1.436206] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
[    1.445874] TCP: Hash tables configured (established 524288 bind 65536)
[    1.452716] TCP reno registered
[    1.466350] NET: Registered protocol family 1
[    1.471023] checking if image is initramfs... it is
[    1.572646] Freeing initrd memory: 1921k freed
[    1.578336] Simple Boot Flag at 0x37 set to 0x80
[    1.585905] audit: initializing netlink socket (disabled)
[    1.591560] type=2000 audit(1221176947.589:1): initialized
[    1.602577] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.613210] VFS: Disk quotas dquot_6.5.1
[    1.617471] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.625780] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    1.632809] msgmni has been set to 15840
[    1.637191] async_tx: api initialized (async)
[    1.641747] io scheduler noop registered
[    1.645871] io scheduler anticipatory registered
[    1.650694] io scheduler deadline registered
[    1.655347] io scheduler cfq registered (default)
[    1.660281] pci 0000:00:00.0: Enabling HT MSI Mapping
[    1.879582] pci 0000:00:05.0: Enabling HT MSI Mapping
[    1.884851] pci 0000:00:05.1: Enabling HT MSI Mapping
[    1.890123] pci 0000:00:05.2: Enabling HT MSI Mapping
[    1.895393] pci 0000:00:06.0: Enabling HT MSI Mapping
[    1.900669] pci 0000:00:08.0: Enabling HT MSI Mapping
[    1.905945] pci 0000:00:09.0: Enabling HT MSI Mapping
[    1.911213] pci 0000:00:0a.0: Enabling HT MSI Mapping
[    1.916486] pci 0000:00:0d.0: Enabling HT MSI Mapping
[    1.921761] pci 0000:00:0f.0: Enabling HT MSI Mapping
[    1.927029] pci 0000:01:04.0: Boot video device
[    1.927212] pcieport-driver 0000:00:0a.0: setting latency timer to 64
[    1.927241] pcieport-driver 0000:00:0a.0: found MSI capability
[    1.933324] pci_express 0000:00:0a.0:pcie00: allocate port service
[    1.933414] pci_express 0000:00:0a.0:pcie01: allocate port service
[    1.933475] pci_express 0000:00:0a.0:pcie03: allocate port service
[    1.933576] pcieport-driver 0000:00:0d.0: setting latency timer to 64
[    1.933604] pcieport-driver 0000:00:0d.0: found MSI capability
[    1.939667] pci_express 0000:00:0d.0:pcie00: allocate port service
[    1.939741] pci_express 0000:00:0d.0:pcie01: allocate port service
[    1.939801] pci_express 0000:00:0d.0:pcie03: allocate port service
[    1.939898] pcieport-driver 0000:00:0f.0: setting latency timer to 64
[    1.939925] pcieport-driver 0000:00:0f.0: found MSI capability
[    1.945977] pci_express 0000:00:0f.0:pcie00: allocate port service
[    1.946048] pci_express 0000:00:0f.0:pcie01: allocate port service
[    1.946108] pci_express 0000:00:0f.0:pcie03: allocate port service
[    1.947934] aer 0000:00:0a.0:pcie01: AER service couldn't init device: no _OSC support
[    1.949554] aer 0000:00:0d.0:pcie01: AER service couldn't init device: no _OSC support
[    1.951148] aer 0000:00:0f.0:pcie01: AER service couldn't init device: no _OSC support
[    1.951606] input: Power Button (FF) as /class/input/input0
[    1.957384] ACPI: Power Button (FF) [PWRF]
[    1.961801] input: Power Button (CM) as /class/input/input1
[    1.967565] ACPI: Power Button (CM) [PWRB]
[    1.972294] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[    1.978363] processor ACPI0007:00: registered as cooling_device0
[    1.984576] ACPI: Processor [C000] (supports 2 throttling states)
[    1.991063] processor ACPI0007:01: registered as cooling_device1
[    1.997366] processor ACPI0007:02: registered as cooling_device2
[    2.003684] processor ACPI0007:03: registered as cooling_device3
[    2.079999] hpet_resources: 0xfed00000 is busy
[    2.080193] Non-volatile memory driver v1.2
[    2.084715] Linux agpgart interface v0.103
[    2.089107] Serial: 8250/16550 driver4 ports, IRQ sharing disabled
[    2.095680] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.102659] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.108789] Floppy drive(s): fd0 is 1.44M
[    2.130069] FDC 0 is a post-1991 82077
[    2.137592] brd: module loaded
[    2.142041] loop: module loaded
[    2.145379] tun: Universal TUN/TAP device driver, 1.6
[    2.150619] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.157181] console [netcon0] enabled
[    2.161035] netconsole: network logging started
[    2.165762] Uniform Multi-Platform E-IDE driver
[    2.170615] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
[    2.180715] Probing IDE interface ide0...
[    3.149712] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive
[    3.206282] Probing IDE interface ide1...
[    3.739617] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[    3.744443] ide1 at 0x170-0x177,0x376 on irq 15
[    3.751258] hdb: ATAPI 24X CD-ROM drive, 256kB Cache
[    3.756646] Uniform CD-ROM driver Revision: 3.20
[    3.771462] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
[    3.779263] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[    3.786512] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008
[    3.792690] Driver 'sd' needs updating - please use bus_type methods
[    3.799315] Driver 'sr' needs updating - please use bus_type methods
[    3.806449] sata_nv 0000:00:05.0: version 3.5
[    3.806741] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23
[    3.812700] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 (level, low) -> IRQ 23
[    3.821311] sata_nv 0000:00:05.0: Using SWNCQ mode
[    3.826430] sata_nv 0000:00:05.0: setting latency timer to 64
[    3.826613] scsi0 : sata_nv
[    3.829796] scsi1 : sata_nv
[    3.832986] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq 23
[    3.840314] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq 23
[    4.179871] ata1: SATA link down (SStatus 0 SControl 300)
[    4.516536] ata2: SATA link down (SStatus 0 SControl 300)
[    4.522429] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22
[    4.528388] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 (level, low) -> IRQ 22
[    4.536997] sata_nv 0000:00:05.1: Using SWNCQ mode
[    4.542083] sata_nv 0000:00:05.1: setting latency timer to 64
[    4.542220] scsi2 : sata_nv
[    4.545334] scsi3 : sata_nv
[    4.548512] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq 22
[    4.555840] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq 22
[    4.896535] ata3: SATA link down (SStatus 0 SControl 300)
[    5.233200] ata4: SATA link down (SStatus 0 SControl 300)
[    5.239087] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21
[    5.245047] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 (level, low) -> IRQ 21
[    5.253654] sata_nv 0000:00:05.2: Using SWNCQ mode
[    5.258744] sata_nv 0000:00:05.2: setting latency timer to 64
[    5.258881] scsi4 : sata_nv
[    5.261998] scsi5 : sata_nv
[    5.265174] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq 21
[    5.272504] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq 21
[    5.613197] ata5: SATA link down (SStatus 0 SControl 300)
[    5.949864] ata6: SATA link down (SStatus 0 SControl 300)
[    5.955612] pata_amd 0000:00:04.0: version 0.3.10
[    5.955640] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region [0x1f0-0x1f7]
[    5.963385] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 (errno=-16)
[    5.971653] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region [0x170-0x177]
[    5.979396] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 (errno=-16)
[    5.987656] pata_amd 0000:00:04.0: no available native port
[    5.993731] Fusion MPT base driver 3.04.07
[    5.998021] Copyright (c) 1999-2008 LSI Corporation
[    6.003109] Fusion MPT SPI Host driver 3.04.07
[    6.007833] Fusion MPT SAS Host driver 3.04.07
[    6.013164] ieee1394: raw1394: /dev/raw1394 device initialized
[    6.019695] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20
[    6.025650] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 (level, low) -> IRQ 20
[    6.034353] ehci_hcd 0000:00:02.1: setting latency timer to 64
[    6.034355] ehci_hcd 0000:00:02.1: EHCI Host Controller
[    6.040002] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
[    6.047776] ehci_hcd 0000:00:02.1: debug port 1
[    6.052503] ehci_hcd 0000:00:02.1: cache line size of 64 is not supported
[    6.052521] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000
[    6.066176] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[    6.074154] usb usb1: configuration #1 chosen from 1 choice
[    6.080016] hub 1-0:1.0: USB hub found
[    6.083976] hub 1-0:1.0: 10 ports detected
[    6.189886] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.190142] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19
[    6.196100] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 (level, low) -> IRQ 19
[    6.204801] ohci_hcd 0000:00:02.0: setting latency timer to 64
[    6.204804] ohci_hcd 0000:00:02.0: OHCI Host Controller
[    6.210376] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
[    6.218132] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000
[    6.278315] usb usb2: configuration #1 chosen from 1 choice
[    6.284165] hub 2-0:1.0: USB hub found
[    6.288118] hub 2-0:1.0: 10 ports detected
[    6.393142] USB Universal Host Controller Interface driver v3.0
[    6.399423] usbcore: registered new interface driver usblp
[    6.405107] Initializing USB Mass Storage driver...
[    6.410243] usbcore: registered new interface driver usb-storage
[    6.416443] USB Mass Storage support registered.
[    6.421406] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12
[    6.679856] serio: i8042 KBD port at 0x60,0x64 irq 1
[    6.685238] mice: PS/2 mouse device common for all mice
[    6.877983] md: linear personality registered for level -1
[    6.883663] md: raid0 personality registered for level 0
[    6.889166] md: raid1 personality registered for level 1
[    6.894669] md: raid10 personality registered for level 10
[    6.956180] raid6: int64x1   2587 MB/s
[    7.016180] raid6: int64x2   3291 MB/s
[    7.076168] raid6: int64x4   3160 MB/s
[    7.136168] raid6: int64x8   2360 MB/s
[    7.196178] raid6: sse2x1    3858 MB/s
[    7.256175] raid6: sse2x2    5139 MB/s
[    7.316176] raid6: sse2x4    5262 MB/s
[    7.320122] raid6: using algorithm sse2x4 (5262 MB/s)
[    7.325365] md: raid6 personality registered for level 6
[    7.335744] md: raid5 personality registered for level 5
[    7.341252] md: raid4 personality registered for level 4
[    7.346753] md: multipath personality registered for level -4
[    7.352960] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
[    7.361877] cpuidle: using governor ladder
[    7.366402] cpuidle: using governor menu
[    7.370591] usbcore: registered new interface driver usbhid
[    7.376362] usbhid: v2.6:USB HID core driver
[    7.380936] oprofile: using NMI interrupt.
[    7.385695] TCP cubic registered
[    7.389332] NET: Registered protocol family 10
[    7.394985] IPv6 over IPv4 tunneling driver
[    7.400135] NET: Registered protocol family 17
[    7.405122] RPC: Registered udp transport module.
[    7.410021] RPC: Registered tcp transport module.
[    7.415002] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 processors (4 cpu cores) (version 2.20.00)
[    7.425975] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
[    7.431831] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
[    7.437687] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
[    7.443537] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
[    7.449301] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
[    7.455149] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
[    7.460999] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
[    7.467050] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
[    7.472929] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
[    7.478785] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
[    7.484637] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
[    7.490411] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
[    7.496279] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
[    7.502134] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
[    7.508739] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    7.515289] Freeing unused kernel memory: 676k freed
[    7.520818] Write protecting the kernel read-only data: 5672k
[    7.742843] Clocksource tsc unstable (delta = -111149082 ns)
[    8.177545] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18
[    8.177654] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 (level, low) -> IRQ 18
[    8.177666] arcmsr 0000:02:00.0: setting latency timer to 64
[    8.189529] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31  
[    8.202850] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 capable)
[    8.202853]  Driver Version 1.20.00.15 2008/02/27
[    8.205018] scsi 6:0:0:0: Direct-Access     Areca    ARC-1280-VOL#00  R001 PQ: 0 ANSI: 5
[    8.207052] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[    8.207117] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
[    8.207182] sd 6:0:0:0: [sda] Write Protect is off
[    8.207188] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
[    8.207298] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.207560] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[    8.207621] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
[    8.207685] sd 6:0:0:0: [sda] Write Protect is off
[    8.207692] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
[    8.207802] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.207811]  sda: sda1 sda2 sda3 sda4
[    8.245400] sd 6:0:0:0: [sda] Attached SCSI disk
[    8.247390] sd 6:0:0:0: Attached scsi generic sg0 type 0
[    8.249720] scsi 6:0:16:0: Processor         Areca    RAID controller  R001 PQ: 0 ANSI: 0
[    8.252259] scsi 6:0:16:0: Attached scsi generic sg1 type 3
[    8.504889] 3ware Storage Controller device driver for Linux v1.26.02.002.
[    8.542592] 3ware 9000 Storage Controller device driver for Linux v2.26.02.011.
[    8.601009] Adaptec aacraid driver 1.1-5[2456]-ms
[    8.993029] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
[    8.998828] SGI XFS Quota Management subsystem
[    9.048982] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
[    9.048990] Copyright (c) 1999-2006 Intel Corporation.
[   12.346739] EXT3-fs: INFO: recovery required on readonly filesystem.
[   12.346748] EXT3-fs: write access will be enabled during recovery.
[   13.161202] kjournald starting.  Commit interval 5 seconds
[   13.161240] EXT3-fs: sda3: orphan cleanup on readonly fs
[   13.161256] ext3_orphan_cleanup: deleting unreferenced inode 2501876
[   13.161508] ext3_orphan_cleanup: deleting unreferenced inode 2501875
[   13.161529] ext3_orphan_cleanup: deleting unreferenced inode 2501874
[   13.161549] ext3_orphan_cleanup: deleting unreferenced inode 2501873
[   13.161565] EXT3-fs: sda3: 4 orphan inodes deleted
[   13.161570] EXT3-fs: recovery complete.
[   13.162085] EXT3-fs: mounted filesystem with ordered data mode.
[   15.634222] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2440
[   15.634257] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2400
[   15.660203] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.61.
[   15.660582] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 17
[   15.660659] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 17 (level, low) -> IRQ 17
[   15.660663] forcedeth 0000:00:08.0: setting latency timer to 64
[   15.663568] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 0, addr 00:e0:81:76:8b:be
[   15.663571] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
[   15.663876] ACPI: PCI Interrupt Link [LMA2] enabled at IRQ 16
[   15.663898] forcedeth 0000:00:09.0: PCI INT A -> Link[LMA2] -> GSI 16 (level, low) -> IRQ 16
[   15.663923] forcedeth 0000:00:09.0: setting latency timer to 64
[   15.665255] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 1, addr 00:e0:81:76:8b:bf
[   15.665257] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
[   18.107517] EXT3 FS on sda3, internal journal
[   18.380032] SMsC 37B787 watchdog component driver 1.1 initialising...
[   18.381424] smsc37b787_wdt: Timeout set to 60 second(s).
[   18.381429] smsc37b787_wdt: Watchdog initialized and sleeping (nowayout=0)...
[   21.417259] kjournald starting.  Commit interval 5 seconds
[   21.417297] EXT3-fs warning: checktime reached, running e2fsck is recommended
[   21.422349] EXT3 FS on sda4, internal journal
[   21.422359] EXT3-fs: recovery complete.
[   21.422430] EXT3-fs: mounted filesystem with ordered data mode.
[   21.501039] Adding 15999992k swap on /dev/sda2.  Priority:-1 extents:1 across:15999992k
[   42.038582] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   42.045903] NFSD: starting 90-second grace period
[   45.396027] eth0: no IPv6 routers present
[   49.761642] w83627hf: Found W83627HF chip at 0xc00

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-13  0:13     ` Joshua Hoblitt
@ 2008-09-15 21:06       ` Joshua Hoblitt
  2008-09-16  2:54         ` Joshua Hoblitt
  0 siblings, 1 reply; 20+ messages in thread
From: Joshua Hoblitt @ 2008-09-15 21:06 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: Andrew Morton, linux-kernel, Thomas Gleixner, bugme-daemon,
	j_kernel

Since Friday 2 different machines have experienced crashes.  One was a total
deadlock with no response on the console.  The other one reported the trace
below on the console and stopped responding to ssh but I was able to loging via
the serial console and reboot the system.  This particular system has had a
number of "odd" kernel traces over the last year and I'm starting to actually
wonder if it may have a bad DIMM in it as occasionally the failure mode seems
to be different then the deadlocks/etc. we see in the other 15 nodes with
identical hardware.

[30712.654542] general protection fault: 0000 [1] SMP 
<Sep/12 09:25 pm>[30712.657678] CPU 3 
<Sep/12 09:25 pm>[30712.657678] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp i2c_nforce2 i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Sep/12 09:25 pm>[30712.657678] Pid: 1178, comm: rpciod/3 Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2
<Sep/12 09:25 pm>[30712.657678] RIP: 0010:[<ffffffff805ac57d>]  [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8
<Sep/12 09:25 pm>[30712.657678] RSP: 0018:ffff88012e5d1e08  EFLAGS: 00010206
<Sep/12 09:25 pm>[30712.657678] RAX: ffffffff807adb48 RBX: ffff880126d61088 RCX: 0400000000000000
<Sep/12 09:25 pm>[30712.657678] RDX: ffff88022bcc0380 RSI: ffff88022bcc0000 RDI: ffff880126d61088
<Sep/12 09:25 pm>[30712.657678] RBP: ffff88022bc88000 R08: 0000000000000003 R09: ffff88022e038f10
<Sep/12 09:25 pm>[30712.657678] R10: 0000000000000001 R11: ffff88012e4a0048 R12: 0400000000000000
<Sep/12 09:25 pm>[30712.657678] R13: ffffffff8059f7a8 R14: ffff88022bc88610 R15: 0000000000000000
<Sep/12 09:25 pm>[30712.657678] FS:  00007f006fb306f0(0000) GS:ffff88022fa0d780(0000) knlGS:0000000000000000

<Sep/12 09:25 pm>[30712.657678] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<Sep/12 09:25 pm>[30712.657678] CR2: 00000000011e3018 CR3: 00000001f3527000 CR4:<Sep/12 09:25 pm>
                 <Sep/12 09:25 pm>[30712.657678] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Sep/12 09:25 pm>[30712.657678] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Sep/12 09:25 pm>[30712.657678] Process rpciod/3 (pid: 1178, threadinfo ffff88012e5d0000, task ffff88012e5acb50)
<Sep/12 09:25 pm>[30712.657678] Stack:  ffffffff8059b996 ffff880126d61088 00000000fffffff5 ffff880126d61118
<Sep/12 09:25 pm>[30712.657678]  ffffffff8059f7a8 ffffffff808a4b20 ffffffff80599aa1 ffffffff8059f7a8
<Sep/12 09:25 pm>[30712.657678]  0000000000000000 ffff880126d61088 ffffffff8059f5f3 ffff88012e5d1e90
<Sep/12 09:25 pm>[30712.657678] Call Trace:
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059b996>] ? xprt_release+0x2c/0x172
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059f7a8>] ? rpc_async_schedule+0x0/0xc
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff80599aa1>] ? call_reserveresult+0x95/0xd1
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059f7a8>] ? rpc_async_schedule+0x0/0xc
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059f5f3>] ? __rpc_execute+0x73/0x228
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff80244394>] ? run_workqueue+0xed/0x1ed
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8024433e>] ? run_workqueue+0x97/0x1ed
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff80244ef7>] ? worker_thread+0xd8/0xe3
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8024791a>] ? autoremove_wake_function+0x0/0x2e
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff80244e1f>] ? worker_thread+0x0/0xe3
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff80247805>] ? kthread+0x47/0x76
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff80230223>] ? schedule_tail+0x27/0x5f
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8020ce09>] ? child_rip+0xa/0x11

<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8024767c>] ? kthreadd+0x167/0x18c
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff802477be>] ? kthread+0x0/0x76
<Sep/12 09:25 pm>[30712.657678]  [<ffffffff8020cdff>] ? child_rip+0x0/0x11
<Sep/12 09:25 pm>[30712.657678] 
<Sep/12 09:25 pm>[30712.657678] 
<Sep/12 09:25 pm>[30712.657678] Code: b0 98 00 00 00 48 85 f6 0f 94 c2 48 85 c9 0f 94 c0 08 c2 0f 85 94 00 00 00 48 8b 47 38 8b 50 28 48 c1 e2 07 48 8d 14 16 48 ff 02 <48> 63 81 50 01 00 00 48 01 42 08 0f b7 87 e0 00 00 00 48 01 42 
<Sep/12 09:25 pm>[30712.657678] RIP  [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8
<Sep/12 09:25 pm>[30712.657678]  RSP <ffff88012e5d1e08>
<Sep/12 09:25 pm>[30712.951578] ---[ end trace e646bef2de7f0b20 ]---

-J

--
On Fri, Sep 12, 2008 at 02:13:06PM -1000, Joshua Hoblitt wrote:
> I just rolled out -rc5 from netdev + Andrew's debug patch + the HPET
> patch Thomas pointed me at.  I'll let it roast on these 14 machine is
> production over the weekend to see if we get another panic.
> 
> I'm attaching the dmesg from this kernel.  We're still getting the NMI
> watchdog warning and the rtc is [still] hosed (I think it was last
> working around -rc3).
> 
> -J
> 
> --
> On Fri, Sep 12, 2008 at 11:13:08AM +0200, Ingo Molnar wrote:
> > 
> > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > > I can't work out who called panic(), nor why.
> > > 
> > > The panic code called the kexec code which called mutex_trylock() 
> > > which called spin_lock_mutex() which then stupidly went and blurted a 
> > > load of debug stuff because of in_interrupt().
> > 
> > agreed - applied your fix in the form below to tip/master - thanks 
> > Andrew.
> > 
> > J, you might want to try tip/master, it includes all known fixes for 
> > this area and this debug improvement as well. You can pick it up via:
> > 
> >   http://people.redhat.com/mingo/tip.git/README
> > 
> > 	Ingo
> > 
> > ---------->
> > >From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001
> > From: Andrew Morton <akpm@linux-foundation.org>
> > Date: Thu, 11 Sep 2008 17:02:58 -0700
> > Subject: [PATCH] lock debug: sit tight when we are already in a panic
> > 
> > in:
> > 
> >   > http://bugzilla.kernel.org/show_bug.cgi?id=11543
> > 
> > The panic code called the kexec code which called mutex_trylock() which
> > called spin_lock_mutex() which then stupidly went and blurted a load of
> > debug stuff because of in_interrupt().
> > 
> > Keep the lock debug code from escallating an already crappy situation.
> > 
> > Signed-off-by: Ingo Molnar <mingo@elte.hu>
> > ---
> >  include/linux/debug_locks.h |    2 +-
> >  1 files changed, 1 insertions(+), 1 deletions(-)
> > 
> > diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h
> > index 4aaa4af..096476f 100644
> > --- a/include/linux/debug_locks.h
> > +++ b/include/linux/debug_locks.h
> > @@ -17,7 +17,7 @@ extern int debug_locks_off(void);
> >  ({									\
> >  	int __ret = 0;							\
> >  									\
> > -	if (unlikely(c)) {						\
> > +	if (!oops_in_progress && unlikely(c)) {				\
> >  		if (debug_locks_off() && !debug_locks_silent)		\
> >  			WARN_ON(1);					\
> >  		__ret = 1;						\

> [    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Linux version 2.6.27-rc4-21704-gd25e26b (root@ipp000) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #1 SMP Tue Sep 2 16:05:45 HST 2008
> [    0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8
> [    0.000000] KERNEL supported cpus:
> [    0.000000]   Intel GenuineIntel
> [    0.000000]   AMD AuthenticAMD
> [    0.000000]   Centaur CentaurHauls
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
> [    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
> [    0.000000]  BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved)
> [    0.000000]  BIOS-e820: 0000000000100000 - 00000000cff60000 (usable)
> [    0.000000]  BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data)
> [    0.000000]  BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS)
> [    0.000000]  BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved)
> [    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
> [    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
> [    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
> [    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
> [    0.000000]  BIOS-e820: 0000000100000000 - 0000000230000000 (usable)
> [    0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff
> [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
> [    0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff
> [    0.000000] init_memory_mapping
> [    0.000000]  0000000000 - 00cfe00000 page 2M
> [    0.000000]  00cfe00000 - 00cff60000 page 4k
> [    0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000
> [    0.000000] last_map_addr: cff60000 end: cff60000
> [    0.000000] init_memory_mapping
> [    0.000000]  0100000000 - 0230000000 page 2M
> [    0.000000] kernel direct mapping tables up to 230000000 @ c000-16000
> [    0.000000] last_map_addr: 230000000 end: 230000000
> [    0.000000] RAMDISK: 37e0f000 - 37fef7dc
> [    0.000000] DMI present.
> [    0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD )
> [    0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD  	 XSDT    6040000  LTP        0)
> [    0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S      6040000 PTL_    F4240)
> [    0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA      CK8  6040000 MSFT  3000000)
> [    0.000000] ACPI: FACS CFF6AFC0, 0040
> [    0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD    POWERNOW  6040000 AMD         1)
> [    0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD    HAMMER    6040000 AMD         1)
> [    0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
> [    0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
> [    0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD  HPETTBL   6040000  LTP        1)
> [    0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD  	 APIC    6040000  LTP        0)
> [    0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
> [    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
> [    0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0
> [    0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1
> [    0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1
> [    0.000000] SRAT: Node 0 PXM 0 0-a0000
> [    0.000000] SRAT: Node 0 PXM 0 100000-d0000000
> [    0.000000] SRAT: Node 0 PXM 0 100000000-130000000
> [    0.000000] SRAT: Node 1 PXM 1 130000000-230000000
> [    0.000000] NUMA: Allocated memnodemap from 11000 - 15680
> [    0.000000] NUMA: Using 20 for the hash shift.
> [    0.000000] Bootmem setup node 0 0000000000000000-0000000130000000
> [    0.000000]   NODE_DATA [0000000000015680 - 000000000002067f]
> [    0.000000]   bootmap [0000000000021000 -  0000000000046fff] pages 26
> [    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000]
> [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
> [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
> [    0.000000]   #2 [0000200000 - 0000f96890]    TEXT DATA BSS ==> [0000200000 - 0000f96890]
> [    0.000000]   #3 [0037e0f000 - 0037fef7dc]          RAMDISK ==> [0037e0f000 - 0037fef7dc]
> [    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
> [    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE ==> [0000008000 - 000000c000]
> [    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE ==> [000000c000 - 0000011000]
> [    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP ==> [0000011000 - 0000015680]
> [    0.000000] Bootmem setup node 1 0000000130000000-0000000230000000
> [    0.000000]   NODE_DATA [0000000130000000 - 000000013000afff]
> [    0.000000]   bootmap [000000013000b000 -  000000013002afff] pages 20
> [    0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000]
> [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page
> [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE
> [    0.000000]   #2 [0000200000 - 0000f96890]    TEXT DATA BSS
> [    0.000000]   #3 [0037e0f000 - 0037fef7dc]          RAMDISK
> [    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved
> [    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE
> [    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE
> [    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP
> [    0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600
> [    0.000000]  [ffffe20000000000-ffffe200071fffff] PMD -> [ffff880028200000-ffff88002e1fffff] on node 0
> [    0.000000]  [ffffe20007200000-ffffe2000d1fffff] PMD -> [ffff880130200000-ffff8801361fffff] on node 1
> [    0.000000] Zone PFN ranges:
> [    0.000000]   DMA      0x00000000 -> 0x00001000
> [    0.000000]   DMA32    0x00001000 -> 0x00100000
> [    0.000000]   Normal   0x00100000 -> 0x00230000
> [    0.000000] Movable zone start PFN for each node
> [    0.000000] early_node_map[4] active PFN ranges
> [    0.000000]     0: 0x00000000 -> 0x0000009d
> [    0.000000]     0: 0x00000100 -> 0x000cff60
> [    0.000000]     0: 0x00100000 -> 0x00130000
> [    0.000000]     1: 0x00130000 -> 0x00230000
> [    0.000000] On node 0 totalpages: 1048317
> [    0.000000]   DMA zone: 307 pages, LIFO batch:0
> [    0.000000]   DMA32 zone: 823232 pages, LIFO batch:31
> [    0.000000]   Normal zone: 192000 pages, LIFO batch:31
> [    0.000000] On node 1 totalpages: 1048576
> [    0.000000]   Normal zone: 1024000 pages, LIFO batch:31
> [    0.000000] Detected use of extended apic ids on hypertransport bus
> [    0.000000] Detected use of extended apic ids on hypertransport bus
> [    0.000000] ACPI: PM-Timer IO Port: 0x1008
> [    0.000000] ACPI: Local APIC address 0xfee00000
> [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
> [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
> [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
> [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
> [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
> [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
> [    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
> [    0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> [    0.000000] ACPI: IRQ0 used by override.
> [    0.000000] ACPI: IRQ2 used by override.
> [    0.000000] ACPI: IRQ9 used by override.
> [    0.000000] Setting APIC routing to flat
> [    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
> [    0.000000] Using ACPI (MADT) for SMP configuration information
> [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
> [    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
> [    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ce000
> [    0.000000] PM: Registered nosave memory: 00000000000ce000 - 0000000000100000
> [    0.000000] PM: Registered nosave memory: 00000000cff60000 - 00000000cff6a000
> [    0.000000] PM: Registered nosave memory: 00000000cff6a000 - 00000000cff80000
> [    0.000000] PM: Registered nosave memory: 00000000cff80000 - 00000000d0000000
> [    0.000000] PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000
> [    0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
> [    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
> [    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000
> [    0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000
> [    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
> [    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fff80000
> [    0.000000] PM: Registered nosave memory: 00000000fff80000 - 0000000100000000
> [    0.000000] Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
> [    0.000000] PERCPU: Allocating 51872 bytes of per cpu data
> [    0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2
> [    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 2039539
> [    0.000000] Policy zone: Normal
> [    0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8
> [    0.000000] Initializing CPU#0
> [    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
> [    0.000000] Extended CMOS year: 2000
> [    0.000000] TSC calibrated against PM_TIMER
> [    0.000000] Detected 2800.009 MHz processor.
> [    0.003333] spurious 8259A interrupt: IRQ7.
> [    0.003333] Console: colour VGA+ 80x25
> [    0.003333] console [tty0] enabled
> [    0.003333] console [ttyS0] enabled
> [    0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> [    0.003333] ... MAX_LOCKDEP_SUBCLASSES:    8
> [    0.003333] ... MAX_LOCK_DEPTH:          48
> [    0.003333] ... MAX_LOCKDEP_KEYS:        8191
> [    0.003333] ... CLASSHASH_SIZE:           4096
> [    0.003333] ... MAX_LOCKDEP_ENTRIES:     8192
> [    0.003333] ... MAX_LOCKDEP_CHAINS:      16384
> [    0.003333] ... CHAINHASH_SIZE:          8192
> [    0.003333]  memory used by lock dependency info: 3839 kB
> [    0.003333]  per task-struct memory footprint: 1920 bytes
> [    0.003333] Checking aperture...
> [    0.003333] No AGP bridge found
> [    0.003333] Node 0: aperture @ 0 size 32 MB
> [    0.003333] Your BIOS doesn't leave a aperture memory hole
> [    0.003333] Please enable the IOMMU option in the BIOS setup
> [    0.003333] This costs you 64 MB of RAM
> [    0.003333] Mapping aperture over 65536 KB of RAM @ 20000000
> [    0.003333] PM: Registered nosave memory: 0000000020000000 - 0000000024000000
> [    0.003333] Memory: 8108304k/9175040k available (3896k kernel code, 279268k reserved, 2243k data, 676k init)
> [    0.003333] CPA: page pool initialized 1 of 1 pages preallocated
> [    0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
> [    0.003333] hpet clockevent registered
> [    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.35 BogoMIPS (lpj=9333363)
> [    0.006698] Security Framework initialized
> [    0.010737] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
> [    0.018937] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
> [    0.025421] Mount-cache hash table entries: 256
> [    0.026974] Initializing cgroup subsys ns
> [    0.030010] Initializing cgroup subsys cpuacct
> [    0.033335] Initializing cgroup subsys memory
> [    0.036674] Initializing cgroup subsys devices
> [    0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> [    0.043332] CPU: L2 Cache: 1024K (64 bytes/line)
> [    0.046666] CPU 0/0 -> Node 0
> [    0.049999] tseg: 00cff80000
> [    0.050000] CPU: Physical Processor ID: 0
> [    0.053331] CPU: Processor Core ID: 0
> [    0.056670] using C1E aware idle routine
> [    0.060018] ACPI: Core revision 20080609
> [    0.070530] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> [    0.108872] activating NMI Watchdog ... done.
> [    0.109994] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> [    0.116660] Using local APIC timer interrupts.
> [    0.123326] APIC timer calibration result 12500076
> [    0.123328] Detected 12.500 MHz APIC timer.
> [    0.126660] APIC timer registered as dummy, due to nmi_watchdog=1!
> [    0.130154] lockdep: fixing up alternatives.
> [    0.133347] Booting processor 1/1 ip 6000
> [    0.003333] Initializing CPU#1
> [    0.003333] Calibrating delay using timer specific routine.. 5602.36 BogoMIPS (lpj=9333385)
> [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> [    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
> [    0.003333] CPU 1/1 -> Node 0
> [    0.003333] CPU: Physical Processor ID: 0
> [    0.003333] CPU: Processor Core ID: 1
> [    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
> [    0.230372] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> [    0.238145] lockdep: fixing up alternatives.
> [    0.239999] Booting processor 2/2 ip 6000
> [    0.003333] Initializing CPU#2
> [    0.003333] Calibrating delay using timer specific routine.. 5602.44 BogoMIPS (lpj=9333513)
> [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> [    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
> [    0.003333] CPU 2/2 -> Node 1
> [    0.003333] CPU: Physical Processor ID: 1
> [    0.003333] CPU: Processor Core ID: 0
> [    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 0x7010600070106
> [    0.340530] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> [    0.348138] lockdep: fixing up alternatives.
> [    0.349997] Booting processor 3/3 ip 6000
> [    0.003333] Initializing CPU#3
> [    0.003333] Calibrating delay using timer specific routine.. 5602.48 BogoMIPS (lpj=9333575)
> [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> [    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
> [    0.003333] CPU 3/3 -> Node 1
> [    0.003333] CPU: Physical Processor ID: 1
> [    0.003333] CPU: Processor Core ID: 1
> [    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106
> [    0.447118] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> [    0.454734] Brought up 4 CPUs
> [    0.456640] Total of 4 processors activated (22409.64 BogoMIPS).
> [    0.460033] Testing NMI watchdog ... 
> [    0.532432] WARNING: CPU#0: NMI appears to be stuck (0->0)!
> [    0.533301] Please report this to bugzilla.kernel.org,
> [    0.536635] and attach the output of the 'dmesg' command.
> [    0.539968] 
> [    0.543302] WARNING: CPU#1: NMI appears to be stuck (0->0)!
> [    0.546635] Please report this to bugzilla.kernel.org,
> [    0.549968] and attach the output of the 'dmesg' command.
> [    0.553301] 
> [    0.554980] WARNING: CPU#2: NMI appears to be stuck (0->0)!
> [    0.556635] Please report this to bugzilla.kernel.org,
> [    0.559968] and attach the output of the 'dmesg' command.
> [    0.563301] 
> [    0.566635] WARNING: CPU#3: NMI appears to be stuck (0->0)!
> [    0.569968] Please report this to bugzilla.kernel.org,
> [    0.573301] and attach the output of the 'dmesg' command.
> [    0.577183] net_namespace: 968 bytes
> [    0.580160] xor: automatically using best checksumming function: generic_sse
> [    0.599969]    generic_sse:  8630.400 MB/sec
> [    0.603302] xor: using function: generic_sse (8630.400 MB/sec)
> [    0.606692] NET: Registered protocol family 16
> [    0.610254] No dock devices found.
> [    0.613503] node 0 link 2: io port [1000, 3fff]
> [    0.613506] TOM: 00000000d0000000 aka 3328M
> [    0.616636] node 0 link 2: mmio [d0000000, dfffffff]
> [    0.616638] node 0 link 2: mmio [a0000, bffff]
> [    0.616640] node 0 link 2: mmio [e0000000, e7ffffff]
> [    0.616643] TOM2: 0000000230000000 aka 8960M
> [    0.619969] bus: [00,ff] on node 0 link 2
> [    0.619970] bus: 00 index 0 io port: [0, ffff]
> [    0.619972] bus: 00 index 1 mmio: [d0000000, ffffffff]
> [    0.619974] bus: 00 index 2 mmio: [a0000, bffff]
> [    0.619975] bus: 00 index 3 mmio: [230000000, fcffffffff]
> [    0.619989] ACPI: bus type pci registered
> [    0.623448] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4
> [    0.626636] PCI: MCFG area at e0000000 reserved in E820
> [    0.630065] PCI: Using MMCONFIG at e0000000 - e04fffff
> [    0.633302] PCI: Using configuration type 1 for base access
> [    0.643980] ACPI: EC: Look up EC in DSDT
> [    0.648409] ACPI: Interpreter enabled
> [    0.649969] ACPI: (supports S0 S1 S3 S4 S5)
> [    0.656635] ACPI: Using IOAPIC for interrupt routing
> [    0.670034] ACPI: PCI Root Bridge [PCI0] (0000:00)
> [    0.673616] pci 0000:00:01.1: PME# supported from D3hot D3cold
> [    0.676644] pci 0000:00:01.1: PME# disabled
> [    0.680009] pci 0000:00:02.0: supports D1
> [    0.680010] pci 0000:00:02.0: supports D2
> [    0.680012] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.683303] pci 0000:00:02.0: PME# disabled
> [    0.686672] pci 0000:00:02.1: supports D1
> [    0.686673] pci 0000:00:02.1: supports D2
> [    0.686675] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.689969] pci 0000:00:02.1: PME# disabled
> [    0.693535] pci 0000:00:08.0: supports D1
> [    0.693537] pci 0000:00:08.0: supports D2
> [    0.693538] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.696637] pci 0000:00:08.0: PME# disabled
> [    0.700017] pci 0000:00:09.0: supports D1
> [    0.700019] pci 0000:00:09.0: supports D2
> [    0.700020] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.703304] pci 0000:00:09.0: PME# disabled
> [    0.706664] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.709969] pci 0000:00:0a.0: PME# disabled
> [    0.713331] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.716636] pci 0000:00:0d.0: PME# disabled
> [    0.719997] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.723302] pci 0000:00:0f.0: PME# disabled
> [    0.726878] pci 0000:01:04.0: supports D1
> [    0.726879] pci 0000:01:04.0: supports D2
> [    0.726904] pci 0000:00:06.0: transparent bridge
> [    0.729969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff]
> [    0.733303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff]
> [    0.736636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff]
> [    0.740034] pci 0000:02:00.0: supports D1
> [    0.740064] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff]
> [    0.743388] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> [    0.743462] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT]
> [    0.743486] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT]
> [    0.743513] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT]
> [    0.743540] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT]
> [    0.769795] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.777165] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> [    0.786790] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.797164] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.808626] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.816220] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> [    0.822660] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.830122] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.840505] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.851950] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> [    0.858235] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> [    0.867922] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 18 19 20 21 22 23)
> [    0.877614] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> [    0.887303] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.898166] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> [    0.909241] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> [    0.915682] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> [    0.921892] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> [    0.928367] Linux Plug and Play Support v0.97 (c) Adam Belay
> [    0.930030] pnp: PnP ACPI init
> [    0.933313] ACPI: bus type pnp registered
> [    0.942967] pnp: PnP ACPI: found 15 devices
> [    0.943302] ACPI: ACPI bus type pnp unregistered
> [    0.946899] SCSI subsystem initialized
> [    0.950132] libata version 3.00 loaded.
> [    0.953483] usbcore: registered new interface driver usbfs
> [    0.956699] usbcore: registered new interface driver hub
> [    0.960060] usbcore: registered new device driver usb
> [    0.963671] PCI: Using ACPI for IRQ routing
> [    0.980227] PCI-DMA: Disabling AGP.
> [    0.984431] PCI-DMA: aperture base @ 20000000 size 65536 KB
> [    0.986635] PCI-DMA: using GART IOMMU.
> [    0.989971] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> [    0.993870] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
> [    1.000341] hpet0: 3 32-bit timers, 25000000 Hz
> [    1.004395] ACPI: RTC can wake from S4
> [    1.006637] Clockevents: could not switch to one-shot mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional.
> [    1.009524] Could not switch to high resolution mode on CPU 3
> [    1.009528] Clockevents: could not switch to one-shot mode: lapic is not functional.
> [    1.009531] Could not switch to high resolution mode on CPU 1
> [    1.009534] Clockevents: could not switch to one-shot mode: lapic is not functional.
> [    1.009538] Could not switch to high resolution mode on CPU 2
> [    1.009969]  lapic is not functional.
> [    1.056944] Could not switch to high resolution mode on CPU 0
> [    1.076221] system 00:00: iomem range 0xffc00000-0xffffffff could not be reserved
> [    1.084054] system 00:00: iomem range 0xfec00000-0xfec00fff could not be reserved
> [    1.091883] system 00:00: iomem range 0xfee00000-0xfeefffff could not be reserved
> [    1.099719] system 00:00: iomem range 0xfed00000-0xfed00fff has been reserved
> [    1.107052] system 00:03: iomem range 0xd0000000-0xd0007fff has been reserved
> [    1.114383] system 00:04: ioport range 0x1000-0x107f has been reserved
> [    1.121101] system 00:04: ioport range 0x1080-0x10ff has been reserved
> [    1.127817] system 00:04: ioport range 0x1400-0x147f has been reserved
> [    1.134537] system 00:04: ioport range 0x1480-0x14ff has been reserved
> [    1.141258] system 00:04: ioport range 0x1800-0x187f has been reserved
> [    1.147976] system 00:04: ioport range 0x1880-0x18ff has been reserved
> [    1.154698] system 00:04: ioport range 0x2440-0x247f has been reserved
> [    1.161420] system 00:04: ioport range 0x2400-0x243f has been reserved
> [    1.168151] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
> [    1.174695] system 00:07: ioport range 0xc05-0xc06 has been reserved
> [    1.182277] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01
> [    1.188388] pci 0000:00:06.0:   IO window: 0x3000-0x3fff
> [    1.193889] pci 0000:00:06.0:   MEM window: 0xd0100000-0xd01fffff
> [    1.200176] pci 0000:00:06.0:   PREFETCH window: 0x000000d8000000-0x000000dfffffff
> [    1.208089] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02
> [    1.214205] pci 0000:00:0a.0:   IO window: disabled
> [    1.219275] pci 0000:00:0a.0:   MEM window: 0xd0200000-0xd02fffff
> [    1.225558] pci 0000:00:0a.0:   PREFETCH window: 0x000000d1000000-0x000000d10fffff
> [    1.233474] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03
> [    1.239598] pci 0000:00:0d.0:   IO window: disabled
> [    1.244666] pci 0000:00:0d.0:   MEM window: disabled
> [    1.249825] pci 0000:00:0d.0:   PREFETCH window: disabled
> [    1.255416] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04
> [    1.261525] pci 0000:00:0f.0:   IO window: disabled
> [    1.266594] pci 0000:00:0f.0:   MEM window: disabled
> [    1.271751] pci 0000:00:0f.0:   PREFETCH window: disabled
> [    1.277347] pci 0000:00:06.0: setting latency timer to 64
> [    1.277352] pci 0000:00:0a.0: setting latency timer to 64
> [    1.277357] pci 0000:00:0d.0: setting latency timer to 64
> [    1.277363] pci 0000:00:0f.0: setting latency timer to 64
> [    1.277365] bus: 00 index 0 io port: [0, ffff]
> [    1.282004] bus: 00 index 1 mmio: [0, ffffffffffffffff]
> [    1.292287] bus: 01 index 0 io port: [3000, 3fff]
> [    1.297187] bus: 01 index 1 mmio: [d0100000, d01fffff]
> [    1.302519] bus: 01 index 2 mmio: [d8000000, dfffffff]
> [    1.307849] bus: 01 index 3 io port: [0, ffff]
> [    1.312483] bus: 01 index 4 mmio: [0, ffffffffffffffff]
> [    1.317900] bus: 02 index 0 mmio: [0, 0]
> [    1.322017] bus: 02 index 1 mmio: [d0200000, d02fffff]
> [    1.327349] bus: 02 index 2 mmio: [d1000000, d10fffff]
> [    1.332678] bus: 02 index 3 mmio: [0, 0]
> [    1.336793] bus: 03 index 0 mmio: [0, 0]
> [    1.340910] bus: 03 index 1 mmio: [0, 0]
> [    1.345025] bus: 03 index 2 mmio: [0, 0]
> [    1.349142] bus: 03 index 3 mmio: [0, 0]
> [    1.353256] bus: 04 index 0 mmio: [0, 0]
> [    1.357374] bus: 04 index 1 mmio: [0, 0]
> [    1.361490] bus: 04 index 2 mmio: [0, 0]
> [    1.365609] bus: 04 index 3 mmio: [0, 0]
> [    1.369739] NET: Registered protocol family 2
> [    1.413137] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
> [    1.423160] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
> [    1.436206] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
> [    1.445874] TCP: Hash tables configured (established 524288 bind 65536)
> [    1.452716] TCP reno registered
> [    1.466350] NET: Registered protocol family 1
> [    1.471023] checking if image is initramfs... it is
> [    1.572646] Freeing initrd memory: 1921k freed
> [    1.578336] Simple Boot Flag at 0x37 set to 0x80
> [    1.585905] audit: initializing netlink socket (disabled)
> [    1.591560] type=2000 audit(1221176947.589:1): initialized
> [    1.602577] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> [    1.613210] VFS: Disk quotas dquot_6.5.1
> [    1.617471] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [    1.625780] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
> [    1.632809] msgmni has been set to 15840
> [    1.637191] async_tx: api initialized (async)
> [    1.641747] io scheduler noop registered
> [    1.645871] io scheduler anticipatory registered
> [    1.650694] io scheduler deadline registered
> [    1.655347] io scheduler cfq registered (default)
> [    1.660281] pci 0000:00:00.0: Enabling HT MSI Mapping
> [    1.879582] pci 0000:00:05.0: Enabling HT MSI Mapping
> [    1.884851] pci 0000:00:05.1: Enabling HT MSI Mapping
> [    1.890123] pci 0000:00:05.2: Enabling HT MSI Mapping
> [    1.895393] pci 0000:00:06.0: Enabling HT MSI Mapping
> [    1.900669] pci 0000:00:08.0: Enabling HT MSI Mapping
> [    1.905945] pci 0000:00:09.0: Enabling HT MSI Mapping
> [    1.911213] pci 0000:00:0a.0: Enabling HT MSI Mapping
> [    1.916486] pci 0000:00:0d.0: Enabling HT MSI Mapping
> [    1.921761] pci 0000:00:0f.0: Enabling HT MSI Mapping
> [    1.927029] pci 0000:01:04.0: Boot video device
> [    1.927212] pcieport-driver 0000:00:0a.0: setting latency timer to 64
> [    1.927241] pcieport-driver 0000:00:0a.0: found MSI capability
> [    1.933324] pci_express 0000:00:0a.0:pcie00: allocate port service
> [    1.933414] pci_express 0000:00:0a.0:pcie01: allocate port service
> [    1.933475] pci_express 0000:00:0a.0:pcie03: allocate port service
> [    1.933576] pcieport-driver 0000:00:0d.0: setting latency timer to 64
> [    1.933604] pcieport-driver 0000:00:0d.0: found MSI capability
> [    1.939667] pci_express 0000:00:0d.0:pcie00: allocate port service
> [    1.939741] pci_express 0000:00:0d.0:pcie01: allocate port service
> [    1.939801] pci_express 0000:00:0d.0:pcie03: allocate port service
> [    1.939898] pcieport-driver 0000:00:0f.0: setting latency timer to 64
> [    1.939925] pcieport-driver 0000:00:0f.0: found MSI capability
> [    1.945977] pci_express 0000:00:0f.0:pcie00: allocate port service
> [    1.946048] pci_express 0000:00:0f.0:pcie01: allocate port service
> [    1.946108] pci_express 0000:00:0f.0:pcie03: allocate port service
> [    1.947934] aer 0000:00:0a.0:pcie01: AER service couldn't init device: no _OSC support
> [    1.949554] aer 0000:00:0d.0:pcie01: AER service couldn't init device: no _OSC support
> [    1.951148] aer 0000:00:0f.0:pcie01: AER service couldn't init device: no _OSC support
> [    1.951606] input: Power Button (FF) as /class/input/input0
> [    1.957384] ACPI: Power Button (FF) [PWRF]
> [    1.961801] input: Power Button (CM) as /class/input/input1
> [    1.967565] ACPI: Power Button (CM) [PWRB]
> [    1.972294] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
> [    1.978363] processor ACPI0007:00: registered as cooling_device0
> [    1.984576] ACPI: Processor [C000] (supports 2 throttling states)
> [    1.991063] processor ACPI0007:01: registered as cooling_device1
> [    1.997366] processor ACPI0007:02: registered as cooling_device2
> [    2.003684] processor ACPI0007:03: registered as cooling_device3
> [    2.079999] hpet_resources: 0xfed00000 is busy
> [    2.080193] Non-volatile memory driver v1.2
> [    2.084715] Linux agpgart interface v0.103
> [    2.089107] Serial: 8250/16550 driver4 ports, IRQ sharing disabled
> [    2.095680] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> [    2.102659] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> [    2.108789] Floppy drive(s): fd0 is 1.44M
> [    2.130069] FDC 0 is a post-1991 82077
> [    2.137592] brd: module loaded
> [    2.142041] loop: module loaded
> [    2.145379] tun: Universal TUN/TAP device driver, 1.6
> [    2.150619] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
> [    2.157181] console [netcon0] enabled
> [    2.161035] netconsole: network logging started
> [    2.165762] Uniform Multi-Platform E-IDE driver
> [    2.170615] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
> [    2.180715] Probing IDE interface ide0...
> [    3.149712] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive
> [    3.206282] Probing IDE interface ide1...
> [    3.739617] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
> [    3.744443] ide1 at 0x170-0x177,0x376 on irq 15
> [    3.751258] hdb: ATAPI 24X CD-ROM drive, 256kB Cache
> [    3.756646] Uniform CD-ROM driver Revision: 3.20
> [    3.771462] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> [    3.779263] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> [    3.786512] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008
> [    3.792690] Driver 'sd' needs updating - please use bus_type methods
> [    3.799315] Driver 'sr' needs updating - please use bus_type methods
> [    3.806449] sata_nv 0000:00:05.0: version 3.5
> [    3.806741] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23
> [    3.812700] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 (level, low) -> IRQ 23
> [    3.821311] sata_nv 0000:00:05.0: Using SWNCQ mode
> [    3.826430] sata_nv 0000:00:05.0: setting latency timer to 64
> [    3.826613] scsi0 : sata_nv
> [    3.829796] scsi1 : sata_nv
> [    3.832986] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq 23
> [    3.840314] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq 23
> [    4.179871] ata1: SATA link down (SStatus 0 SControl 300)
> [    4.516536] ata2: SATA link down (SStatus 0 SControl 300)
> [    4.522429] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22
> [    4.528388] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 (level, low) -> IRQ 22
> [    4.536997] sata_nv 0000:00:05.1: Using SWNCQ mode
> [    4.542083] sata_nv 0000:00:05.1: setting latency timer to 64
> [    4.542220] scsi2 : sata_nv
> [    4.545334] scsi3 : sata_nv
> [    4.548512] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq 22
> [    4.555840] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq 22
> [    4.896535] ata3: SATA link down (SStatus 0 SControl 300)
> [    5.233200] ata4: SATA link down (SStatus 0 SControl 300)
> [    5.239087] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21
> [    5.245047] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 (level, low) -> IRQ 21
> [    5.253654] sata_nv 0000:00:05.2: Using SWNCQ mode
> [    5.258744] sata_nv 0000:00:05.2: setting latency timer to 64
> [    5.258881] scsi4 : sata_nv
> [    5.261998] scsi5 : sata_nv
> [    5.265174] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq 21
> [    5.272504] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq 21
> [    5.613197] ata5: SATA link down (SStatus 0 SControl 300)
> [    5.949864] ata6: SATA link down (SStatus 0 SControl 300)
> [    5.955612] pata_amd 0000:00:04.0: version 0.3.10
> [    5.955640] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region [0x1f0-0x1f7]
> [    5.963385] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 (errno=-16)
> [    5.971653] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region [0x170-0x177]
> [    5.979396] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 (errno=-16)
> [    5.987656] pata_amd 0000:00:04.0: no available native port
> [    5.993731] Fusion MPT base driver 3.04.07
> [    5.998021] Copyright (c) 1999-2008 LSI Corporation
> [    6.003109] Fusion MPT SPI Host driver 3.04.07
> [    6.007833] Fusion MPT SAS Host driver 3.04.07
> [    6.013164] ieee1394: raw1394: /dev/raw1394 device initialized
> [    6.019695] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20
> [    6.025650] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 (level, low) -> IRQ 20
> [    6.034353] ehci_hcd 0000:00:02.1: setting latency timer to 64
> [    6.034355] ehci_hcd 0000:00:02.1: EHCI Host Controller
> [    6.040002] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
> [    6.047776] ehci_hcd 0000:00:02.1: debug port 1
> [    6.052503] ehci_hcd 0000:00:02.1: cache line size of 64 is not supported
> [    6.052521] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000
> [    6.066176] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> [    6.074154] usb usb1: configuration #1 chosen from 1 choice
> [    6.080016] hub 1-0:1.0: USB hub found
> [    6.083976] hub 1-0:1.0: 10 ports detected
> [    6.189886] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
> [    6.190142] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19
> [    6.196100] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 (level, low) -> IRQ 19
> [    6.204801] ohci_hcd 0000:00:02.0: setting latency timer to 64
> [    6.204804] ohci_hcd 0000:00:02.0: OHCI Host Controller
> [    6.210376] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
> [    6.218132] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000
> [    6.278315] usb usb2: configuration #1 chosen from 1 choice
> [    6.284165] hub 2-0:1.0: USB hub found
> [    6.288118] hub 2-0:1.0: 10 ports detected
> [    6.393142] USB Universal Host Controller Interface driver v3.0
> [    6.399423] usbcore: registered new interface driver usblp
> [    6.405107] Initializing USB Mass Storage driver...
> [    6.410243] usbcore: registered new interface driver usb-storage
> [    6.416443] USB Mass Storage support registered.
> [    6.421406] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12
> [    6.679856] serio: i8042 KBD port at 0x60,0x64 irq 1
> [    6.685238] mice: PS/2 mouse device common for all mice
> [    6.877983] md: linear personality registered for level -1
> [    6.883663] md: raid0 personality registered for level 0
> [    6.889166] md: raid1 personality registered for level 1
> [    6.894669] md: raid10 personality registered for level 10
> [    6.956180] raid6: int64x1   2587 MB/s
> [    7.016180] raid6: int64x2   3291 MB/s
> [    7.076168] raid6: int64x4   3160 MB/s
> [    7.136168] raid6: int64x8   2360 MB/s
> [    7.196178] raid6: sse2x1    3858 MB/s
> [    7.256175] raid6: sse2x2    5139 MB/s
> [    7.316176] raid6: sse2x4    5262 MB/s
> [    7.320122] raid6: using algorithm sse2x4 (5262 MB/s)
> [    7.325365] md: raid6 personality registered for level 6
> [    7.335744] md: raid5 personality registered for level 5
> [    7.341252] md: raid4 personality registered for level 4
> [    7.346753] md: multipath personality registered for level -4
> [    7.352960] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
> [    7.361877] cpuidle: using governor ladder
> [    7.366402] cpuidle: using governor menu
> [    7.370591] usbcore: registered new interface driver usbhid
> [    7.376362] usbhid: v2.6:USB HID core driver
> [    7.380936] oprofile: using NMI interrupt.
> [    7.385695] TCP cubic registered
> [    7.389332] NET: Registered protocol family 10
> [    7.394985] IPv6 over IPv4 tunneling driver
> [    7.400135] NET: Registered protocol family 17
> [    7.405122] RPC: Registered udp transport module.
> [    7.410021] RPC: Registered tcp transport module.
> [    7.415002] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 processors (4 cpu cores) (version 2.20.00)
> [    7.425975] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
> [    7.431831] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
> [    7.437687] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
> [    7.443537] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
> [    7.449301] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
> [    7.455149] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
> [    7.460999] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
> [    7.467050] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
> [    7.472929] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
> [    7.478785] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
> [    7.484637] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
> [    7.490411] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
> [    7.496279] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
> [    7.502134] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
> [    7.508739] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
> [    7.515289] Freeing unused kernel memory: 676k freed
> [    7.520818] Write protecting the kernel read-only data: 5672k
> [    7.742843] Clocksource tsc unstable (delta = -111149082 ns)
> [    8.177545] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18
> [    8.177654] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 (level, low) -> IRQ 18
> [    8.177666] arcmsr 0000:02:00.0: setting latency timer to 64
> [    8.189529] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31  
> [    8.202850] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 capable)
> [    8.202853]  Driver Version 1.20.00.15 2008/02/27
> [    8.205018] scsi 6:0:0:0: Direct-Access     Areca    ARC-1280-VOL#00  R001 PQ: 0 ANSI: 5
> [    8.207052] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
> [    8.207117] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
> [    8.207182] sd 6:0:0:0: [sda] Write Protect is off
> [    8.207188] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
> [    8.207298] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [    8.207560] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
> [    8.207621] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
> [    8.207685] sd 6:0:0:0: [sda] Write Protect is off
> [    8.207692] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
> [    8.207802] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [    8.207811]  sda: sda1 sda2 sda3 sda4
> [    8.245400] sd 6:0:0:0: [sda] Attached SCSI disk
> [    8.247390] sd 6:0:0:0: Attached scsi generic sg0 type 0
> [    8.249720] scsi 6:0:16:0: Processor         Areca    RAID controller  R001 PQ: 0 ANSI: 0
> [    8.252259] scsi 6:0:16:0: Attached scsi generic sg1 type 3
> [    8.504889] 3ware Storage Controller device driver for Linux v1.26.02.002.
> [    8.542592] 3ware 9000 Storage Controller device driver for Linux v2.26.02.011.
> [    8.601009] Adaptec aacraid driver 1.1-5[2456]-ms
> [    8.993029] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
> [    8.998828] SGI XFS Quota Management subsystem
> [    9.048982] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
> [    9.048990] Copyright (c) 1999-2006 Intel Corporation.
> [   12.346739] EXT3-fs: INFO: recovery required on readonly filesystem.
> [   12.346748] EXT3-fs: write access will be enabled during recovery.
> [   13.161202] kjournald starting.  Commit interval 5 seconds
> [   13.161240] EXT3-fs: sda3: orphan cleanup on readonly fs
> [   13.161256] ext3_orphan_cleanup: deleting unreferenced inode 2501876
> [   13.161508] ext3_orphan_cleanup: deleting unreferenced inode 2501875
> [   13.161529] ext3_orphan_cleanup: deleting unreferenced inode 2501874
> [   13.161549] ext3_orphan_cleanup: deleting unreferenced inode 2501873
> [   13.161565] EXT3-fs: sda3: 4 orphan inodes deleted
> [   13.161570] EXT3-fs: recovery complete.
> [   13.162085] EXT3-fs: mounted filesystem with ordered data mode.
> [   15.634222] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2440
> [   15.634257] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2400
> [   15.660203] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.61.
> [   15.660582] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 17
> [   15.660659] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 17 (level, low) -> IRQ 17
> [   15.660663] forcedeth 0000:00:08.0: setting latency timer to 64
> [   15.663568] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 0, addr 00:e0:81:76:8b:be
> [   15.663571] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
> [   15.663876] ACPI: PCI Interrupt Link [LMA2] enabled at IRQ 16
> [   15.663898] forcedeth 0000:00:09.0: PCI INT A -> Link[LMA2] -> GSI 16 (level, low) -> IRQ 16
> [   15.663923] forcedeth 0000:00:09.0: setting latency timer to 64
> [   15.665255] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 1, addr 00:e0:81:76:8b:bf
> [   15.665257] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
> [   18.107517] EXT3 FS on sda3, internal journal
> [   18.380032] SMsC 37B787 watchdog component driver 1.1 initialising...
> [   18.381424] smsc37b787_wdt: Timeout set to 60 second(s).
> [   18.381429] smsc37b787_wdt: Watchdog initialized and sleeping (nowayout=0)...
> [   21.417259] kjournald starting.  Commit interval 5 seconds
> [   21.417297] EXT3-fs warning: checktime reached, running e2fsck is recommended
> [   21.422349] EXT3 FS on sda4, internal journal
> [   21.422359] EXT3-fs: recovery complete.
> [   21.422430] EXT3-fs: mounted filesystem with ordered data mode.
> [   21.501039] Adding 15999992k swap on /dev/sda2.  Priority:-1 extents:1 across:15999992k
> [   42.038582] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> [   42.045903] NFSD: starting 90-second grace period
> [   45.396027] eth0: no IPv6 routers present
> [   49.761642] w83627hf: Found W83627HF chip at 0xc00


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-15 21:06       ` Joshua Hoblitt
@ 2008-09-16  2:54         ` Joshua Hoblitt
  2008-09-16 14:14           ` Thomas Gleixner
  0 siblings, 1 reply; 20+ messages in thread
From: Joshua Hoblitt @ 2008-09-16  2:54 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: Andrew Morton, linux-kernel, Thomas Gleixner, bugme-daemon,
	j_kernel

In addition to the deadlocks, we still have the watchdog warning:

[    0.460034] Testing NMI watchdog ... 
[    0.532557] WARNING: CPU#0: NMI appears to be stuck (0->0)!
[    0.533301] Please report this to bugzilla.kernel.org,
[    0.536635] and attach the output of the 'dmesg' command.

Perhaps an HPET problem:

[    0.993800] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
[    0.999969] hpet0: 3 32-bit timers, 25000000 Hz
[    1.004396] ACPI: RTC can wake from S4
[    1.006637] Clockevents: could not switch to one-shot
mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional.
[    1.009844] Could not switch to high resolution mode on CPU 3
[    1.009848] Clockevents: could not switch to one-shot mode: lapic is not functional.
[    1.009852] Could not switch to high resolution mode on CPU 2
[    1.009855] Clockevents: could not switch to one-shot mode: lapic is not functional.
[    1.009858] Could not switch to high resolution mode on CPU 1
[    1.009969]  lapic is not functional.
[    1.056944] Could not switch to high resolution mode on CPU 0

And a failure to create a /dev/rtc[0] device with udev 115 or 119 and
this entry in the dmesg.

[    7.498900] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)

-J

--
On Mon, Sep 15, 2008 at 11:06:35AM -1000, Joshua Hoblitt wrote:
> Since Friday 2 different machines have experienced crashes.  One was a total
> deadlock with no response on the console.  The other one reported the trace
> below on the console and stopped responding to ssh but I was able to loging via
> the serial console and reboot the system.  This particular system has had a
> number of "odd" kernel traces over the last year and I'm starting to actually
> wonder if it may have a bad DIMM in it as occasionally the failure mode seems
> to be different then the deadlocks/etc. we see in the other 15 nodes with
> identical hardware.
> 
> [30712.654542] general protection fault: 0000 [1] SMP 
> <Sep/12 09:25 pm>[30712.657678] CPU 3 
> <Sep/12 09:25 pm>[30712.657678] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp i2c_nforce2 i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
> <Sep/12 09:25 pm>[30712.657678] Pid: 1178, comm: rpciod/3 Not tainted 2.6.27-rc5-22033-gd26acd9-dirty #2
> <Sep/12 09:25 pm>[30712.657678] RIP: 0010:[<ffffffff805ac57d>]  [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8
> <Sep/12 09:25 pm>[30712.657678] RSP: 0018:ffff88012e5d1e08  EFLAGS: 00010206
> <Sep/12 09:25 pm>[30712.657678] RAX: ffffffff807adb48 RBX: ffff880126d61088 RCX: 0400000000000000
> <Sep/12 09:25 pm>[30712.657678] RDX: ffff88022bcc0380 RSI: ffff88022bcc0000 RDI: ffff880126d61088
> <Sep/12 09:25 pm>[30712.657678] RBP: ffff88022bc88000 R08: 0000000000000003 R09: ffff88022e038f10
> <Sep/12 09:25 pm>[30712.657678] R10: 0000000000000001 R11: ffff88012e4a0048 R12: 0400000000000000
> <Sep/12 09:25 pm>[30712.657678] R13: ffffffff8059f7a8 R14: ffff88022bc88610 R15: 0000000000000000
> <Sep/12 09:25 pm>[30712.657678] FS:  00007f006fb306f0(0000) GS:ffff88022fa0d780(0000) knlGS:0000000000000000
> 
> <Sep/12 09:25 pm>[30712.657678] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> <Sep/12 09:25 pm>[30712.657678] CR2: 00000000011e3018 CR3: 00000001f3527000 CR4:<Sep/12 09:25 pm>
>                  <Sep/12 09:25 pm>[30712.657678] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> <Sep/12 09:25 pm>[30712.657678] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> <Sep/12 09:25 pm>[30712.657678] Process rpciod/3 (pid: 1178, threadinfo ffff88012e5d0000, task ffff88012e5acb50)
> <Sep/12 09:25 pm>[30712.657678] Stack:  ffffffff8059b996 ffff880126d61088 00000000fffffff5 ffff880126d61118
> <Sep/12 09:25 pm>[30712.657678]  ffffffff8059f7a8 ffffffff808a4b20 ffffffff80599aa1 ffffffff8059f7a8
> <Sep/12 09:25 pm>[30712.657678]  0000000000000000 ffff880126d61088 ffffffff8059f5f3 ffff88012e5d1e90
> <Sep/12 09:25 pm>[30712.657678] Call Trace:
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059b996>] ? xprt_release+0x2c/0x172
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059f7a8>] ? rpc_async_schedule+0x0/0xc
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff80599aa1>] ? call_reserveresult+0x95/0xd1
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059f7a8>] ? rpc_async_schedule+0x0/0xc
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8059f5f3>] ? __rpc_execute+0x73/0x228
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff80244394>] ? run_workqueue+0xed/0x1ed
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8024433e>] ? run_workqueue+0x97/0x1ed
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff80244ef7>] ? worker_thread+0xd8/0xe3
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8024791a>] ? autoremove_wake_function+0x0/0x2e
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff80244e1f>] ? worker_thread+0x0/0xe3
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff80247805>] ? kthread+0x47/0x76
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff80230223>] ? schedule_tail+0x27/0x5f
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8020ce09>] ? child_rip+0xa/0x11
> 
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8024767c>] ? kthreadd+0x167/0x18c
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff802477be>] ? kthread+0x0/0x76
> <Sep/12 09:25 pm>[30712.657678]  [<ffffffff8020cdff>] ? child_rip+0x0/0x11
> <Sep/12 09:25 pm>[30712.657678] 
> <Sep/12 09:25 pm>[30712.657678] 
> <Sep/12 09:25 pm>[30712.657678] Code: b0 98 00 00 00 48 85 f6 0f 94 c2 48 85 c9 0f 94 c0 08 c2 0f 85 94 00 00 00 48 8b 47 38 8b 50 28 48 c1 e2 07 48 8d 14 16 48 ff 02 <48> 63 81 50 01 00 00 48 01 42 08 0f b7 87 e0 00 00 00 48 01 42 
> <Sep/12 09:25 pm>[30712.657678] RIP  [<ffffffff805ac57d>] rpc_count_iostats+0x35/0xb8
> <Sep/12 09:25 pm>[30712.657678]  RSP <ffff88012e5d1e08>
> <Sep/12 09:25 pm>[30712.951578] ---[ end trace e646bef2de7f0b20 ]---
> 
> -J
> 
> --
> On Fri, Sep 12, 2008 at 02:13:06PM -1000, Joshua Hoblitt wrote:
> > I just rolled out -rc5 from netdev + Andrew's debug patch + the HPET
> > patch Thomas pointed me at.  I'll let it roast on these 14 machine is
> > production over the weekend to see if we get another panic.
> > 
> > I'm attaching the dmesg from this kernel.  We're still getting the NMI
> > watchdog warning and the rtc is [still] hosed (I think it was last
> > working around -rc3).
> > 
> > -J
> > 
> > --
> > On Fri, Sep 12, 2008 at 11:13:08AM +0200, Ingo Molnar wrote:
> > > 
> > > * Andrew Morton <akpm@linux-foundation.org> wrote:
> > > 
> > > > I can't work out who called panic(), nor why.
> > > > 
> > > > The panic code called the kexec code which called mutex_trylock() 
> > > > which called spin_lock_mutex() which then stupidly went and blurted a 
> > > > load of debug stuff because of in_interrupt().
> > > 
> > > agreed - applied your fix in the form below to tip/master - thanks 
> > > Andrew.
> > > 
> > > J, you might want to try tip/master, it includes all known fixes for 
> > > this area and this debug improvement as well. You can pick it up via:
> > > 
> > >   http://people.redhat.com/mingo/tip.git/README
> > > 
> > > 	Ingo
> > > 
> > > ---------->
> > > >From 53b9d87f41a3d8838210ad7cdef02d814817ce85 Mon Sep 17 00:00:00 2001
> > > From: Andrew Morton <akpm@linux-foundation.org>
> > > Date: Thu, 11 Sep 2008 17:02:58 -0700
> > > Subject: [PATCH] lock debug: sit tight when we are already in a panic
> > > 
> > > in:
> > > 
> > >   > http://bugzilla.kernel.org/show_bug.cgi?id=11543
> > > 
> > > The panic code called the kexec code which called mutex_trylock() which
> > > called spin_lock_mutex() which then stupidly went and blurted a load of
> > > debug stuff because of in_interrupt().
> > > 
> > > Keep the lock debug code from escallating an already crappy situation.
> > > 
> > > Signed-off-by: Ingo Molnar <mingo@elte.hu>
> > > ---
> > >  include/linux/debug_locks.h |    2 +-
> > >  1 files changed, 1 insertions(+), 1 deletions(-)
> > > 
> > > diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h
> > > index 4aaa4af..096476f 100644
> > > --- a/include/linux/debug_locks.h
> > > +++ b/include/linux/debug_locks.h
> > > @@ -17,7 +17,7 @@ extern int debug_locks_off(void);
> > >  ({									\
> > >  	int __ret = 0;							\
> > >  									\
> > > -	if (unlikely(c)) {						\
> > > +	if (!oops_in_progress && unlikely(c)) {				\
> > >  		if (debug_locks_off() && !debug_locks_silent)		\
> > >  			WARN_ON(1);					\
> > >  		__ret = 1;						\
> 
> > [    0.000000] Initializing cgroup subsys cpuset
> > [    0.000000] Linux version 2.6.27-rc4-21704-gd25e26b (root@ipp000) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #1 SMP Tue Sep 2 16:05:45 HST 2008
> > [    0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8
> > [    0.000000] KERNEL supported cpus:
> > [    0.000000]   Intel GenuineIntel
> > [    0.000000]   AMD AuthenticAMD
> > [    0.000000]   Centaur CentaurHauls
> > [    0.000000] BIOS-provided physical RAM map:
> > [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
> > [    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
> > [    0.000000]  BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved)
> > [    0.000000]  BIOS-e820: 0000000000100000 - 00000000cff60000 (usable)
> > [    0.000000]  BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data)
> > [    0.000000]  BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS)
> > [    0.000000]  BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved)
> > [    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
> > [    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
> > [    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
> > [    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
> > [    0.000000]  BIOS-e820: 0000000100000000 - 0000000230000000 (usable)
> > [    0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff
> > [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
> > [    0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff
> > [    0.000000] init_memory_mapping
> > [    0.000000]  0000000000 - 00cfe00000 page 2M
> > [    0.000000]  00cfe00000 - 00cff60000 page 4k
> > [    0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000
> > [    0.000000] last_map_addr: cff60000 end: cff60000
> > [    0.000000] init_memory_mapping
> > [    0.000000]  0100000000 - 0230000000 page 2M
> > [    0.000000] kernel direct mapping tables up to 230000000 @ c000-16000
> > [    0.000000] last_map_addr: 230000000 end: 230000000
> > [    0.000000] RAMDISK: 37e0f000 - 37fef7dc
> > [    0.000000] DMI present.
> > [    0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD )
> > [    0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD  	 XSDT    6040000  LTP        0)
> > [    0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S      6040000 PTL_    F4240)
> > [    0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA      CK8  6040000 MSFT  3000000)
> > [    0.000000] ACPI: FACS CFF6AFC0, 0040
> > [    0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD    POWERNOW  6040000 AMD         1)
> > [    0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD    HAMMER    6040000 AMD         1)
> > [    0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
> > [    0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
> > [    0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD  HPETTBL   6040000  LTP        1)
> > [    0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD  	 APIC    6040000  LTP        0)
> > [    0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
> > [    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
> > [    0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0
> > [    0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1
> > [    0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1
> > [    0.000000] SRAT: Node 0 PXM 0 0-a0000
> > [    0.000000] SRAT: Node 0 PXM 0 100000-d0000000
> > [    0.000000] SRAT: Node 0 PXM 0 100000000-130000000
> > [    0.000000] SRAT: Node 1 PXM 1 130000000-230000000
> > [    0.000000] NUMA: Allocated memnodemap from 11000 - 15680
> > [    0.000000] NUMA: Using 20 for the hash shift.
> > [    0.000000] Bootmem setup node 0 0000000000000000-0000000130000000
> > [    0.000000]   NODE_DATA [0000000000015680 - 000000000002067f]
> > [    0.000000]   bootmap [0000000000021000 -  0000000000046fff] pages 26
> > [    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000]
> > [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
> > [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
> > [    0.000000]   #2 [0000200000 - 0000f96890]    TEXT DATA BSS ==> [0000200000 - 0000f96890]
> > [    0.000000]   #3 [0037e0f000 - 0037fef7dc]          RAMDISK ==> [0037e0f000 - 0037fef7dc]
> > [    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
> > [    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE ==> [0000008000 - 000000c000]
> > [    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE ==> [000000c000 - 0000011000]
> > [    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP ==> [0000011000 - 0000015680]
> > [    0.000000] Bootmem setup node 1 0000000130000000-0000000230000000
> > [    0.000000]   NODE_DATA [0000000130000000 - 000000013000afff]
> > [    0.000000]   bootmap [000000013000b000 -  000000013002afff] pages 20
> > [    0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000]
> > [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page
> > [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE
> > [    0.000000]   #2 [0000200000 - 0000f96890]    TEXT DATA BSS
> > [    0.000000]   #3 [0037e0f000 - 0037fef7dc]          RAMDISK
> > [    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved
> > [    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE
> > [    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE
> > [    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP
> > [    0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600
> > [    0.000000]  [ffffe20000000000-ffffe200071fffff] PMD -> [ffff880028200000-ffff88002e1fffff] on node 0
> > [    0.000000]  [ffffe20007200000-ffffe2000d1fffff] PMD -> [ffff880130200000-ffff8801361fffff] on node 1
> > [    0.000000] Zone PFN ranges:
> > [    0.000000]   DMA      0x00000000 -> 0x00001000
> > [    0.000000]   DMA32    0x00001000 -> 0x00100000
> > [    0.000000]   Normal   0x00100000 -> 0x00230000
> > [    0.000000] Movable zone start PFN for each node
> > [    0.000000] early_node_map[4] active PFN ranges
> > [    0.000000]     0: 0x00000000 -> 0x0000009d
> > [    0.000000]     0: 0x00000100 -> 0x000cff60
> > [    0.000000]     0: 0x00100000 -> 0x00130000
> > [    0.000000]     1: 0x00130000 -> 0x00230000
> > [    0.000000] On node 0 totalpages: 1048317
> > [    0.000000]   DMA zone: 307 pages, LIFO batch:0
> > [    0.000000]   DMA32 zone: 823232 pages, LIFO batch:31
> > [    0.000000]   Normal zone: 192000 pages, LIFO batch:31
> > [    0.000000] On node 1 totalpages: 1048576
> > [    0.000000]   Normal zone: 1024000 pages, LIFO batch:31
> > [    0.000000] Detected use of extended apic ids on hypertransport bus
> > [    0.000000] Detected use of extended apic ids on hypertransport bus
> > [    0.000000] ACPI: PM-Timer IO Port: 0x1008
> > [    0.000000] ACPI: Local APIC address 0xfee00000
> > [    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> > [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
> > [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
> > [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
> > [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
> > [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
> > [    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
> > [    0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23
> > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
> > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> > [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > [    0.000000] ACPI: IRQ0 used by override.
> > [    0.000000] ACPI: IRQ2 used by override.
> > [    0.000000] ACPI: IRQ9 used by override.
> > [    0.000000] Setting APIC routing to flat
> > [    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
> > [    0.000000] Using ACPI (MADT) for SMP configuration information
> > [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
> > [    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
> > [    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
> > [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ce000
> > [    0.000000] PM: Registered nosave memory: 00000000000ce000 - 0000000000100000
> > [    0.000000] PM: Registered nosave memory: 00000000cff60000 - 00000000cff6a000
> > [    0.000000] PM: Registered nosave memory: 00000000cff6a000 - 00000000cff80000
> > [    0.000000] PM: Registered nosave memory: 00000000cff80000 - 00000000d0000000
> > [    0.000000] PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000
> > [    0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
> > [    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
> > [    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000
> > [    0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000
> > [    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
> > [    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fff80000
> > [    0.000000] PM: Registered nosave memory: 00000000fff80000 - 0000000100000000
> > [    0.000000] Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
> > [    0.000000] PERCPU: Allocating 51872 bytes of per cpu data
> > [    0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2
> > [    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 2039539
> > [    0.000000] Policy zone: Normal
> > [    0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=1 show_msr=1 console=tty0 console=ttyS0,115200n8
> > [    0.000000] Initializing CPU#0
> > [    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
> > [    0.000000] Extended CMOS year: 2000
> > [    0.000000] TSC calibrated against PM_TIMER
> > [    0.000000] Detected 2800.009 MHz processor.
> > [    0.003333] spurious 8259A interrupt: IRQ7.
> > [    0.003333] Console: colour VGA+ 80x25
> > [    0.003333] console [tty0] enabled
> > [    0.003333] console [ttyS0] enabled
> > [    0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> > [    0.003333] ... MAX_LOCKDEP_SUBCLASSES:    8
> > [    0.003333] ... MAX_LOCK_DEPTH:          48
> > [    0.003333] ... MAX_LOCKDEP_KEYS:        8191
> > [    0.003333] ... CLASSHASH_SIZE:           4096
> > [    0.003333] ... MAX_LOCKDEP_ENTRIES:     8192
> > [    0.003333] ... MAX_LOCKDEP_CHAINS:      16384
> > [    0.003333] ... CHAINHASH_SIZE:          8192
> > [    0.003333]  memory used by lock dependency info: 3839 kB
> > [    0.003333]  per task-struct memory footprint: 1920 bytes
> > [    0.003333] Checking aperture...
> > [    0.003333] No AGP bridge found
> > [    0.003333] Node 0: aperture @ 0 size 32 MB
> > [    0.003333] Your BIOS doesn't leave a aperture memory hole
> > [    0.003333] Please enable the IOMMU option in the BIOS setup
> > [    0.003333] This costs you 64 MB of RAM
> > [    0.003333] Mapping aperture over 65536 KB of RAM @ 20000000
> > [    0.003333] PM: Registered nosave memory: 0000000020000000 - 0000000024000000
> > [    0.003333] Memory: 8108304k/9175040k available (3896k kernel code, 279268k reserved, 2243k data, 676k init)
> > [    0.003333] CPA: page pool initialized 1 of 1 pages preallocated
> > [    0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
> > [    0.003333] hpet clockevent registered
> > [    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.35 BogoMIPS (lpj=9333363)
> > [    0.006698] Security Framework initialized
> > [    0.010737] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
> > [    0.018937] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
> > [    0.025421] Mount-cache hash table entries: 256
> > [    0.026974] Initializing cgroup subsys ns
> > [    0.030010] Initializing cgroup subsys cpuacct
> > [    0.033335] Initializing cgroup subsys memory
> > [    0.036674] Initializing cgroup subsys devices
> > [    0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> > [    0.043332] CPU: L2 Cache: 1024K (64 bytes/line)
> > [    0.046666] CPU 0/0 -> Node 0
> > [    0.049999] tseg: 00cff80000
> > [    0.050000] CPU: Physical Processor ID: 0
> > [    0.053331] CPU: Processor Core ID: 0
> > [    0.056670] using C1E aware idle routine
> > [    0.060018] ACPI: Core revision 20080609
> > [    0.070530] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> > [    0.108872] activating NMI Watchdog ... done.
> > [    0.109994] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> > [    0.116660] Using local APIC timer interrupts.
> > [    0.123326] APIC timer calibration result 12500076
> > [    0.123328] Detected 12.500 MHz APIC timer.
> > [    0.126660] APIC timer registered as dummy, due to nmi_watchdog=1!
> > [    0.130154] lockdep: fixing up alternatives.
> > [    0.133347] Booting processor 1/1 ip 6000
> > [    0.003333] Initializing CPU#1
> > [    0.003333] Calibrating delay using timer specific routine.. 5602.36 BogoMIPS (lpj=9333385)
> > [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> > [    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
> > [    0.003333] CPU 1/1 -> Node 0
> > [    0.003333] CPU: Physical Processor ID: 0
> > [    0.003333] CPU: Processor Core ID: 1
> > [    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
> > [    0.230372] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> > [    0.238145] lockdep: fixing up alternatives.
> > [    0.239999] Booting processor 2/2 ip 6000
> > [    0.003333] Initializing CPU#2
> > [    0.003333] Calibrating delay using timer specific routine.. 5602.44 BogoMIPS (lpj=9333513)
> > [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> > [    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
> > [    0.003333] CPU 2/2 -> Node 1
> > [    0.003333] CPU: Physical Processor ID: 1
> > [    0.003333] CPU: Processor Core ID: 0
> > [    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 0x7010600070106
> > [    0.340530] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> > [    0.348138] lockdep: fixing up alternatives.
> > [    0.349997] Booting processor 3/3 ip 6000
> > [    0.003333] Initializing CPU#3
> > [    0.003333] Calibrating delay using timer specific routine.. 5602.48 BogoMIPS (lpj=9333575)
> > [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> > [    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
> > [    0.003333] CPU 3/3 -> Node 1
> > [    0.003333] CPU: Physical Processor ID: 1
> > [    0.003333] CPU: Processor Core ID: 1
> > [    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106
> > [    0.447118] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
> > [    0.454734] Brought up 4 CPUs
> > [    0.456640] Total of 4 processors activated (22409.64 BogoMIPS).
> > [    0.460033] Testing NMI watchdog ... 
> > [    0.532432] WARNING: CPU#0: NMI appears to be stuck (0->0)!
> > [    0.533301] Please report this to bugzilla.kernel.org,
> > [    0.536635] and attach the output of the 'dmesg' command.
> > [    0.539968] 
> > [    0.543302] WARNING: CPU#1: NMI appears to be stuck (0->0)!
> > [    0.546635] Please report this to bugzilla.kernel.org,
> > [    0.549968] and attach the output of the 'dmesg' command.
> > [    0.553301] 
> > [    0.554980] WARNING: CPU#2: NMI appears to be stuck (0->0)!
> > [    0.556635] Please report this to bugzilla.kernel.org,
> > [    0.559968] and attach the output of the 'dmesg' command.
> > [    0.563301] 
> > [    0.566635] WARNING: CPU#3: NMI appears to be stuck (0->0)!
> > [    0.569968] Please report this to bugzilla.kernel.org,
> > [    0.573301] and attach the output of the 'dmesg' command.
> > [    0.577183] net_namespace: 968 bytes
> > [    0.580160] xor: automatically using best checksumming function: generic_sse
> > [    0.599969]    generic_sse:  8630.400 MB/sec
> > [    0.603302] xor: using function: generic_sse (8630.400 MB/sec)
> > [    0.606692] NET: Registered protocol family 16
> > [    0.610254] No dock devices found.
> > [    0.613503] node 0 link 2: io port [1000, 3fff]
> > [    0.613506] TOM: 00000000d0000000 aka 3328M
> > [    0.616636] node 0 link 2: mmio [d0000000, dfffffff]
> > [    0.616638] node 0 link 2: mmio [a0000, bffff]
> > [    0.616640] node 0 link 2: mmio [e0000000, e7ffffff]
> > [    0.616643] TOM2: 0000000230000000 aka 8960M
> > [    0.619969] bus: [00,ff] on node 0 link 2
> > [    0.619970] bus: 00 index 0 io port: [0, ffff]
> > [    0.619972] bus: 00 index 1 mmio: [d0000000, ffffffff]
> > [    0.619974] bus: 00 index 2 mmio: [a0000, bffff]
> > [    0.619975] bus: 00 index 3 mmio: [230000000, fcffffffff]
> > [    0.619989] ACPI: bus type pci registered
> > [    0.623448] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4
> > [    0.626636] PCI: MCFG area at e0000000 reserved in E820
> > [    0.630065] PCI: Using MMCONFIG at e0000000 - e04fffff
> > [    0.633302] PCI: Using configuration type 1 for base access
> > [    0.643980] ACPI: EC: Look up EC in DSDT
> > [    0.648409] ACPI: Interpreter enabled
> > [    0.649969] ACPI: (supports S0 S1 S3 S4 S5)
> > [    0.656635] ACPI: Using IOAPIC for interrupt routing
> > [    0.670034] ACPI: PCI Root Bridge [PCI0] (0000:00)
> > [    0.673616] pci 0000:00:01.1: PME# supported from D3hot D3cold
> > [    0.676644] pci 0000:00:01.1: PME# disabled
> > [    0.680009] pci 0000:00:02.0: supports D1
> > [    0.680010] pci 0000:00:02.0: supports D2
> > [    0.680012] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.683303] pci 0000:00:02.0: PME# disabled
> > [    0.686672] pci 0000:00:02.1: supports D1
> > [    0.686673] pci 0000:00:02.1: supports D2
> > [    0.686675] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.689969] pci 0000:00:02.1: PME# disabled
> > [    0.693535] pci 0000:00:08.0: supports D1
> > [    0.693537] pci 0000:00:08.0: supports D2
> > [    0.693538] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.696637] pci 0000:00:08.0: PME# disabled
> > [    0.700017] pci 0000:00:09.0: supports D1
> > [    0.700019] pci 0000:00:09.0: supports D2
> > [    0.700020] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.703304] pci 0000:00:09.0: PME# disabled
> > [    0.706664] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.709969] pci 0000:00:0a.0: PME# disabled
> > [    0.713331] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.716636] pci 0000:00:0d.0: PME# disabled
> > [    0.719997] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.723302] pci 0000:00:0f.0: PME# disabled
> > [    0.726878] pci 0000:01:04.0: supports D1
> > [    0.726879] pci 0000:01:04.0: supports D2
> > [    0.726904] pci 0000:00:06.0: transparent bridge
> > [    0.729969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff]
> > [    0.733303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff]
> > [    0.736636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff]
> > [    0.740034] pci 0000:02:00.0: supports D1
> > [    0.740064] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff]
> > [    0.743388] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> > [    0.743462] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT]
> > [    0.743486] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT]
> > [    0.743513] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT]
> > [    0.743540] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT]
> > [    0.769795] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.777165] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> > [    0.786790] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.797164] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.808626] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.816220] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> > [    0.822660] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.830122] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.840505] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.851950] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> > [    0.858235] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> > [    0.867922] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 18 19 20 21 22 23)
> > [    0.877614] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> > [    0.887303] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.898166] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
> > [    0.909241] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> > [    0.915682] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
> > [    0.921892] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
> > [    0.928367] Linux Plug and Play Support v0.97 (c) Adam Belay
> > [    0.930030] pnp: PnP ACPI init
> > [    0.933313] ACPI: bus type pnp registered
> > [    0.942967] pnp: PnP ACPI: found 15 devices
> > [    0.943302] ACPI: ACPI bus type pnp unregistered
> > [    0.946899] SCSI subsystem initialized
> > [    0.950132] libata version 3.00 loaded.
> > [    0.953483] usbcore: registered new interface driver usbfs
> > [    0.956699] usbcore: registered new interface driver hub
> > [    0.960060] usbcore: registered new device driver usb
> > [    0.963671] PCI: Using ACPI for IRQ routing
> > [    0.980227] PCI-DMA: Disabling AGP.
> > [    0.984431] PCI-DMA: aperture base @ 20000000 size 65536 KB
> > [    0.986635] PCI-DMA: using GART IOMMU.
> > [    0.989971] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> > [    0.993870] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
> > [    1.000341] hpet0: 3 32-bit timers, 25000000 Hz
> > [    1.004395] ACPI: RTC can wake from S4
> > [    1.006637] Clockevents: could not switch to one-shot mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional.
> > [    1.009524] Could not switch to high resolution mode on CPU 3
> > [    1.009528] Clockevents: could not switch to one-shot mode: lapic is not functional.
> > [    1.009531] Could not switch to high resolution mode on CPU 1
> > [    1.009534] Clockevents: could not switch to one-shot mode: lapic is not functional.
> > [    1.009538] Could not switch to high resolution mode on CPU 2
> > [    1.009969]  lapic is not functional.
> > [    1.056944] Could not switch to high resolution mode on CPU 0
> > [    1.076221] system 00:00: iomem range 0xffc00000-0xffffffff could not be reserved
> > [    1.084054] system 00:00: iomem range 0xfec00000-0xfec00fff could not be reserved
> > [    1.091883] system 00:00: iomem range 0xfee00000-0xfeefffff could not be reserved
> > [    1.099719] system 00:00: iomem range 0xfed00000-0xfed00fff has been reserved
> > [    1.107052] system 00:03: iomem range 0xd0000000-0xd0007fff has been reserved
> > [    1.114383] system 00:04: ioport range 0x1000-0x107f has been reserved
> > [    1.121101] system 00:04: ioport range 0x1080-0x10ff has been reserved
> > [    1.127817] system 00:04: ioport range 0x1400-0x147f has been reserved
> > [    1.134537] system 00:04: ioport range 0x1480-0x14ff has been reserved
> > [    1.141258] system 00:04: ioport range 0x1800-0x187f has been reserved
> > [    1.147976] system 00:04: ioport range 0x1880-0x18ff has been reserved
> > [    1.154698] system 00:04: ioport range 0x2440-0x247f has been reserved
> > [    1.161420] system 00:04: ioport range 0x2400-0x243f has been reserved
> > [    1.168151] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
> > [    1.174695] system 00:07: ioport range 0xc05-0xc06 has been reserved
> > [    1.182277] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01
> > [    1.188388] pci 0000:00:06.0:   IO window: 0x3000-0x3fff
> > [    1.193889] pci 0000:00:06.0:   MEM window: 0xd0100000-0xd01fffff
> > [    1.200176] pci 0000:00:06.0:   PREFETCH window: 0x000000d8000000-0x000000dfffffff
> > [    1.208089] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02
> > [    1.214205] pci 0000:00:0a.0:   IO window: disabled
> > [    1.219275] pci 0000:00:0a.0:   MEM window: 0xd0200000-0xd02fffff
> > [    1.225558] pci 0000:00:0a.0:   PREFETCH window: 0x000000d1000000-0x000000d10fffff
> > [    1.233474] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03
> > [    1.239598] pci 0000:00:0d.0:   IO window: disabled
> > [    1.244666] pci 0000:00:0d.0:   MEM window: disabled
> > [    1.249825] pci 0000:00:0d.0:   PREFETCH window: disabled
> > [    1.255416] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04
> > [    1.261525] pci 0000:00:0f.0:   IO window: disabled
> > [    1.266594] pci 0000:00:0f.0:   MEM window: disabled
> > [    1.271751] pci 0000:00:0f.0:   PREFETCH window: disabled
> > [    1.277347] pci 0000:00:06.0: setting latency timer to 64
> > [    1.277352] pci 0000:00:0a.0: setting latency timer to 64
> > [    1.277357] pci 0000:00:0d.0: setting latency timer to 64
> > [    1.277363] pci 0000:00:0f.0: setting latency timer to 64
> > [    1.277365] bus: 00 index 0 io port: [0, ffff]
> > [    1.282004] bus: 00 index 1 mmio: [0, ffffffffffffffff]
> > [    1.292287] bus: 01 index 0 io port: [3000, 3fff]
> > [    1.297187] bus: 01 index 1 mmio: [d0100000, d01fffff]
> > [    1.302519] bus: 01 index 2 mmio: [d8000000, dfffffff]
> > [    1.307849] bus: 01 index 3 io port: [0, ffff]
> > [    1.312483] bus: 01 index 4 mmio: [0, ffffffffffffffff]
> > [    1.317900] bus: 02 index 0 mmio: [0, 0]
> > [    1.322017] bus: 02 index 1 mmio: [d0200000, d02fffff]
> > [    1.327349] bus: 02 index 2 mmio: [d1000000, d10fffff]
> > [    1.332678] bus: 02 index 3 mmio: [0, 0]
> > [    1.336793] bus: 03 index 0 mmio: [0, 0]
> > [    1.340910] bus: 03 index 1 mmio: [0, 0]
> > [    1.345025] bus: 03 index 2 mmio: [0, 0]
> > [    1.349142] bus: 03 index 3 mmio: [0, 0]
> > [    1.353256] bus: 04 index 0 mmio: [0, 0]
> > [    1.357374] bus: 04 index 1 mmio: [0, 0]
> > [    1.361490] bus: 04 index 2 mmio: [0, 0]
> > [    1.365609] bus: 04 index 3 mmio: [0, 0]
> > [    1.369739] NET: Registered protocol family 2
> > [    1.413137] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
> > [    1.423160] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
> > [    1.436206] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
> > [    1.445874] TCP: Hash tables configured (established 524288 bind 65536)
> > [    1.452716] TCP reno registered
> > [    1.466350] NET: Registered protocol family 1
> > [    1.471023] checking if image is initramfs... it is
> > [    1.572646] Freeing initrd memory: 1921k freed
> > [    1.578336] Simple Boot Flag at 0x37 set to 0x80
> > [    1.585905] audit: initializing netlink socket (disabled)
> > [    1.591560] type=2000 audit(1221176947.589:1): initialized
> > [    1.602577] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> > [    1.613210] VFS: Disk quotas dquot_6.5.1
> > [    1.617471] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> > [    1.625780] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
> > [    1.632809] msgmni has been set to 15840
> > [    1.637191] async_tx: api initialized (async)
> > [    1.641747] io scheduler noop registered
> > [    1.645871] io scheduler anticipatory registered
> > [    1.650694] io scheduler deadline registered
> > [    1.655347] io scheduler cfq registered (default)
> > [    1.660281] pci 0000:00:00.0: Enabling HT MSI Mapping
> > [    1.879582] pci 0000:00:05.0: Enabling HT MSI Mapping
> > [    1.884851] pci 0000:00:05.1: Enabling HT MSI Mapping
> > [    1.890123] pci 0000:00:05.2: Enabling HT MSI Mapping
> > [    1.895393] pci 0000:00:06.0: Enabling HT MSI Mapping
> > [    1.900669] pci 0000:00:08.0: Enabling HT MSI Mapping
> > [    1.905945] pci 0000:00:09.0: Enabling HT MSI Mapping
> > [    1.911213] pci 0000:00:0a.0: Enabling HT MSI Mapping
> > [    1.916486] pci 0000:00:0d.0: Enabling HT MSI Mapping
> > [    1.921761] pci 0000:00:0f.0: Enabling HT MSI Mapping
> > [    1.927029] pci 0000:01:04.0: Boot video device
> > [    1.927212] pcieport-driver 0000:00:0a.0: setting latency timer to 64
> > [    1.927241] pcieport-driver 0000:00:0a.0: found MSI capability
> > [    1.933324] pci_express 0000:00:0a.0:pcie00: allocate port service
> > [    1.933414] pci_express 0000:00:0a.0:pcie01: allocate port service
> > [    1.933475] pci_express 0000:00:0a.0:pcie03: allocate port service
> > [    1.933576] pcieport-driver 0000:00:0d.0: setting latency timer to 64
> > [    1.933604] pcieport-driver 0000:00:0d.0: found MSI capability
> > [    1.939667] pci_express 0000:00:0d.0:pcie00: allocate port service
> > [    1.939741] pci_express 0000:00:0d.0:pcie01: allocate port service
> > [    1.939801] pci_express 0000:00:0d.0:pcie03: allocate port service
> > [    1.939898] pcieport-driver 0000:00:0f.0: setting latency timer to 64
> > [    1.939925] pcieport-driver 0000:00:0f.0: found MSI capability
> > [    1.945977] pci_express 0000:00:0f.0:pcie00: allocate port service
> > [    1.946048] pci_express 0000:00:0f.0:pcie01: allocate port service
> > [    1.946108] pci_express 0000:00:0f.0:pcie03: allocate port service
> > [    1.947934] aer 0000:00:0a.0:pcie01: AER service couldn't init device: no _OSC support
> > [    1.949554] aer 0000:00:0d.0:pcie01: AER service couldn't init device: no _OSC support
> > [    1.951148] aer 0000:00:0f.0:pcie01: AER service couldn't init device: no _OSC support
> > [    1.951606] input: Power Button (FF) as /class/input/input0
> > [    1.957384] ACPI: Power Button (FF) [PWRF]
> > [    1.961801] input: Power Button (CM) as /class/input/input1
> > [    1.967565] ACPI: Power Button (CM) [PWRB]
> > [    1.972294] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
> > [    1.978363] processor ACPI0007:00: registered as cooling_device0
> > [    1.984576] ACPI: Processor [C000] (supports 2 throttling states)
> > [    1.991063] processor ACPI0007:01: registered as cooling_device1
> > [    1.997366] processor ACPI0007:02: registered as cooling_device2
> > [    2.003684] processor ACPI0007:03: registered as cooling_device3
> > [    2.079999] hpet_resources: 0xfed00000 is busy
> > [    2.080193] Non-volatile memory driver v1.2
> > [    2.084715] Linux agpgart interface v0.103
> > [    2.089107] Serial: 8250/16550 driver4 ports, IRQ sharing disabled
> > [    2.095680] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> > [    2.102659] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> > [    2.108789] Floppy drive(s): fd0 is 1.44M
> > [    2.130069] FDC 0 is a post-1991 82077
> > [    2.137592] brd: module loaded
> > [    2.142041] loop: module loaded
> > [    2.145379] tun: Universal TUN/TAP device driver, 1.6
> > [    2.150619] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
> > [    2.157181] console [netcon0] enabled
> > [    2.161035] netconsole: network logging started
> > [    2.165762] Uniform Multi-Platform E-IDE driver
> > [    2.170615] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
> > [    2.180715] Probing IDE interface ide0...
> > [    3.149712] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive
> > [    3.206282] Probing IDE interface ide1...
> > [    3.739617] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
> > [    3.744443] ide1 at 0x170-0x177,0x376 on irq 15
> > [    3.751258] hdb: ATAPI 24X CD-ROM drive, 256kB Cache
> > [    3.756646] Uniform CD-ROM driver Revision: 3.20
> > [    3.771462] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> > [    3.779263] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> > [    3.786512] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008
> > [    3.792690] Driver 'sd' needs updating - please use bus_type methods
> > [    3.799315] Driver 'sr' needs updating - please use bus_type methods
> > [    3.806449] sata_nv 0000:00:05.0: version 3.5
> > [    3.806741] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23
> > [    3.812700] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 (level, low) -> IRQ 23
> > [    3.821311] sata_nv 0000:00:05.0: Using SWNCQ mode
> > [    3.826430] sata_nv 0000:00:05.0: setting latency timer to 64
> > [    3.826613] scsi0 : sata_nv
> > [    3.829796] scsi1 : sata_nv
> > [    3.832986] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq 23
> > [    3.840314] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq 23
> > [    4.179871] ata1: SATA link down (SStatus 0 SControl 300)
> > [    4.516536] ata2: SATA link down (SStatus 0 SControl 300)
> > [    4.522429] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22
> > [    4.528388] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 (level, low) -> IRQ 22
> > [    4.536997] sata_nv 0000:00:05.1: Using SWNCQ mode
> > [    4.542083] sata_nv 0000:00:05.1: setting latency timer to 64
> > [    4.542220] scsi2 : sata_nv
> > [    4.545334] scsi3 : sata_nv
> > [    4.548512] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq 22
> > [    4.555840] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq 22
> > [    4.896535] ata3: SATA link down (SStatus 0 SControl 300)
> > [    5.233200] ata4: SATA link down (SStatus 0 SControl 300)
> > [    5.239087] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21
> > [    5.245047] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 (level, low) -> IRQ 21
> > [    5.253654] sata_nv 0000:00:05.2: Using SWNCQ mode
> > [    5.258744] sata_nv 0000:00:05.2: setting latency timer to 64
> > [    5.258881] scsi4 : sata_nv
> > [    5.261998] scsi5 : sata_nv
> > [    5.265174] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq 21
> > [    5.272504] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq 21
> > [    5.613197] ata5: SATA link down (SStatus 0 SControl 300)
> > [    5.949864] ata6: SATA link down (SStatus 0 SControl 300)
> > [    5.955612] pata_amd 0000:00:04.0: version 0.3.10
> > [    5.955640] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region [0x1f0-0x1f7]
> > [    5.963385] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 (errno=-16)
> > [    5.971653] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region [0x170-0x177]
> > [    5.979396] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 (errno=-16)
> > [    5.987656] pata_amd 0000:00:04.0: no available native port
> > [    5.993731] Fusion MPT base driver 3.04.07
> > [    5.998021] Copyright (c) 1999-2008 LSI Corporation
> > [    6.003109] Fusion MPT SPI Host driver 3.04.07
> > [    6.007833] Fusion MPT SAS Host driver 3.04.07
> > [    6.013164] ieee1394: raw1394: /dev/raw1394 device initialized
> > [    6.019695] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20
> > [    6.025650] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 (level, low) -> IRQ 20
> > [    6.034353] ehci_hcd 0000:00:02.1: setting latency timer to 64
> > [    6.034355] ehci_hcd 0000:00:02.1: EHCI Host Controller
> > [    6.040002] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
> > [    6.047776] ehci_hcd 0000:00:02.1: debug port 1
> > [    6.052503] ehci_hcd 0000:00:02.1: cache line size of 64 is not supported
> > [    6.052521] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000
> > [    6.066176] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> > [    6.074154] usb usb1: configuration #1 chosen from 1 choice
> > [    6.080016] hub 1-0:1.0: USB hub found
> > [    6.083976] hub 1-0:1.0: 10 ports detected
> > [    6.189886] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
> > [    6.190142] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19
> > [    6.196100] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 (level, low) -> IRQ 19
> > [    6.204801] ohci_hcd 0000:00:02.0: setting latency timer to 64
> > [    6.204804] ohci_hcd 0000:00:02.0: OHCI Host Controller
> > [    6.210376] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
> > [    6.218132] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000
> > [    6.278315] usb usb2: configuration #1 chosen from 1 choice
> > [    6.284165] hub 2-0:1.0: USB hub found
> > [    6.288118] hub 2-0:1.0: 10 ports detected
> > [    6.393142] USB Universal Host Controller Interface driver v3.0
> > [    6.399423] usbcore: registered new interface driver usblp
> > [    6.405107] Initializing USB Mass Storage driver...
> > [    6.410243] usbcore: registered new interface driver usb-storage
> > [    6.416443] USB Mass Storage support registered.
> > [    6.421406] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12
> > [    6.679856] serio: i8042 KBD port at 0x60,0x64 irq 1
> > [    6.685238] mice: PS/2 mouse device common for all mice
> > [    6.877983] md: linear personality registered for level -1
> > [    6.883663] md: raid0 personality registered for level 0
> > [    6.889166] md: raid1 personality registered for level 1
> > [    6.894669] md: raid10 personality registered for level 10
> > [    6.956180] raid6: int64x1   2587 MB/s
> > [    7.016180] raid6: int64x2   3291 MB/s
> > [    7.076168] raid6: int64x4   3160 MB/s
> > [    7.136168] raid6: int64x8   2360 MB/s
> > [    7.196178] raid6: sse2x1    3858 MB/s
> > [    7.256175] raid6: sse2x2    5139 MB/s
> > [    7.316176] raid6: sse2x4    5262 MB/s
> > [    7.320122] raid6: using algorithm sse2x4 (5262 MB/s)
> > [    7.325365] md: raid6 personality registered for level 6
> > [    7.335744] md: raid5 personality registered for level 5
> > [    7.341252] md: raid4 personality registered for level 4
> > [    7.346753] md: multipath personality registered for level -4
> > [    7.352960] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
> > [    7.361877] cpuidle: using governor ladder
> > [    7.366402] cpuidle: using governor menu
> > [    7.370591] usbcore: registered new interface driver usbhid
> > [    7.376362] usbhid: v2.6:USB HID core driver
> > [    7.380936] oprofile: using NMI interrupt.
> > [    7.385695] TCP cubic registered
> > [    7.389332] NET: Registered protocol family 10
> > [    7.394985] IPv6 over IPv4 tunneling driver
> > [    7.400135] NET: Registered protocol family 17
> > [    7.405122] RPC: Registered udp transport module.
> > [    7.410021] RPC: Registered tcp transport module.
> > [    7.415002] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 processors (4 cpu cores) (version 2.20.00)
> > [    7.425975] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
> > [    7.431831] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
> > [    7.437687] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
> > [    7.443537] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
> > [    7.449301] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
> > [    7.455149] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
> > [    7.460999] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
> > [    7.467050] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
> > [    7.472929] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
> > [    7.478785] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
> > [    7.484637] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
> > [    7.490411] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
> > [    7.496279] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
> > [    7.502134] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
> > [    7.508739] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
> > [    7.515289] Freeing unused kernel memory: 676k freed
> > [    7.520818] Write protecting the kernel read-only data: 5672k
> > [    7.742843] Clocksource tsc unstable (delta = -111149082 ns)
> > [    8.177545] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18
> > [    8.177654] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 (level, low) -> IRQ 18
> > [    8.177666] arcmsr 0000:02:00.0: setting latency timer to 64
> > [    8.189529] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31  
> > [    8.202850] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 capable)
> > [    8.202853]  Driver Version 1.20.00.15 2008/02/27
> > [    8.205018] scsi 6:0:0:0: Direct-Access     Areca    ARC-1280-VOL#00  R001 PQ: 0 ANSI: 5
> > [    8.207052] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
> > [    8.207117] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
> > [    8.207182] sd 6:0:0:0: [sda] Write Protect is off
> > [    8.207188] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
> > [    8.207298] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [    8.207560] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
> > [    8.207621] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
> > [    8.207685] sd 6:0:0:0: [sda] Write Protect is off
> > [    8.207692] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
> > [    8.207802] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [    8.207811]  sda: sda1 sda2 sda3 sda4
> > [    8.245400] sd 6:0:0:0: [sda] Attached SCSI disk
> > [    8.247390] sd 6:0:0:0: Attached scsi generic sg0 type 0
> > [    8.249720] scsi 6:0:16:0: Processor         Areca    RAID controller  R001 PQ: 0 ANSI: 0
> > [    8.252259] scsi 6:0:16:0: Attached scsi generic sg1 type 3
> > [    8.504889] 3ware Storage Controller device driver for Linux v1.26.02.002.
> > [    8.542592] 3ware 9000 Storage Controller device driver for Linux v2.26.02.011.
> > [    8.601009] Adaptec aacraid driver 1.1-5[2456]-ms
> > [    8.993029] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
> > [    8.998828] SGI XFS Quota Management subsystem
> > [    9.048982] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
> > [    9.048990] Copyright (c) 1999-2006 Intel Corporation.
> > [   12.346739] EXT3-fs: INFO: recovery required on readonly filesystem.
> > [   12.346748] EXT3-fs: write access will be enabled during recovery.
> > [   13.161202] kjournald starting.  Commit interval 5 seconds
> > [   13.161240] EXT3-fs: sda3: orphan cleanup on readonly fs
> > [   13.161256] ext3_orphan_cleanup: deleting unreferenced inode 2501876
> > [   13.161508] ext3_orphan_cleanup: deleting unreferenced inode 2501875
> > [   13.161529] ext3_orphan_cleanup: deleting unreferenced inode 2501874
> > [   13.161549] ext3_orphan_cleanup: deleting unreferenced inode 2501873
> > [   13.161565] EXT3-fs: sda3: 4 orphan inodes deleted
> > [   13.161570] EXT3-fs: recovery complete.
> > [   13.162085] EXT3-fs: mounted filesystem with ordered data mode.
> > [   15.634222] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2440
> > [   15.634257] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2400
> > [   15.660203] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.61.
> > [   15.660582] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 17
> > [   15.660659] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 17 (level, low) -> IRQ 17
> > [   15.660663] forcedeth 0000:00:08.0: setting latency timer to 64
> > [   15.663568] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 0, addr 00:e0:81:76:8b:be
> > [   15.663571] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
> > [   15.663876] ACPI: PCI Interrupt Link [LMA2] enabled at IRQ 16
> > [   15.663898] forcedeth 0000:00:09.0: PCI INT A -> Link[LMA2] -> GSI 16 (level, low) -> IRQ 16
> > [   15.663923] forcedeth 0000:00:09.0: setting latency timer to 64
> > [   15.665255] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 1, addr 00:e0:81:76:8b:bf
> > [   15.665257] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
> > [   18.107517] EXT3 FS on sda3, internal journal
> > [   18.380032] SMsC 37B787 watchdog component driver 1.1 initialising...
> > [   18.381424] smsc37b787_wdt: Timeout set to 60 second(s).
> > [   18.381429] smsc37b787_wdt: Watchdog initialized and sleeping (nowayout=0)...
> > [   21.417259] kjournald starting.  Commit interval 5 seconds
> > [   21.417297] EXT3-fs warning: checktime reached, running e2fsck is recommended
> > [   21.422349] EXT3 FS on sda4, internal journal
> > [   21.422359] EXT3-fs: recovery complete.
> > [   21.422430] EXT3-fs: mounted filesystem with ordered data mode.
> > [   21.501039] Adding 15999992k swap on /dev/sda2.  Priority:-1 extents:1 across:15999992k
> > [   42.038582] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> > [   42.045903] NFSD: starting 90-second grace period
> > [   45.396027] eth0: no IPv6 routers present
> > [   49.761642] w83627hf: Found W83627HF chip at 0xc00
> 

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-16  2:54         ` Joshua Hoblitt
@ 2008-09-16 14:14           ` Thomas Gleixner
  2008-09-16 17:56             ` Cyrill Gorcunov
  0 siblings, 1 reply; 20+ messages in thread
From: Thomas Gleixner @ 2008-09-16 14:14 UTC (permalink / raw)
  To: Joshua Hoblitt
  Cc: Ingo Molnar, Andrew Morton, linux-kernel, bugme-daemon, j_kernel

On Mon, 15 Sep 2008, Joshua Hoblitt wrote:

> In addition to the deadlocks, we still have the watchdog warning:
> 
> [    0.460034] Testing NMI watchdog ... 
> [    0.532557] WARNING: CPU#0: NMI appears to be stuck (0->0)!
> [    0.533301] Please report this to bugzilla.kernel.org,
> [    0.536635] and attach the output of the 'dmesg' command.
> 
> Perhaps an HPET problem:
> 
> [    0.993800] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
> [    0.999969] hpet0: 3 32-bit timers, 25000000 Hz
> [    1.004396] ACPI: RTC can wake from S4
> [    1.006637] Clockevents: could not switch to one-shot
> mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional.
> [    1.009844] Could not switch to high resolution mode on CPU 3
> [    1.009848] Clockevents: could not switch to one-shot mode: lapic is not functional.
> [    1.009852] Could not switch to high resolution mode on CPU 2
> [    1.009855] Clockevents: could not switch to one-shot mode: lapic is not functional.
> [    1.009858] Could not switch to high resolution mode on CPU 1
> [    1.009969]  lapic is not functional.
> [    1.056944] Could not switch to high resolution mode on CPU 0

No, that's documented behaviour:

> > > [    0.126660] APIC timer registered as dummy, due to nmi_watchdog=1!

Can you try nmi_watchdog=2 ?

Thanks,

	tglx

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-16 14:14           ` Thomas Gleixner
@ 2008-09-16 17:56             ` Cyrill Gorcunov
  2008-09-16 17:57               ` Cyrill Gorcunov
  0 siblings, 1 reply; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-16 17:56 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Joshua Hoblitt, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon, j_kernel

[Thomas Gleixner - Tue, Sep 16, 2008 at 07:14:40AM -0700]
| On Mon, 15 Sep 2008, Joshua Hoblitt wrote:
| 
| > In addition to the deadlocks, we still have the watchdog warning:
| > 
| > [    0.460034] Testing NMI watchdog ... 
| > [    0.532557] WARNING: CPU#0: NMI appears to be stuck (0->0)!
| > [    0.533301] Please report this to bugzilla.kernel.org,
| > [    0.536635] and attach the output of the 'dmesg' command.
| > 
| > Perhaps an HPET problem:
| > 
| > [    0.993800] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
| > [    0.999969] hpet0: 3 32-bit timers, 25000000 Hz
| > [    1.004396] ACPI: RTC can wake from S4
| > [    1.006637] Clockevents: could not switch to one-shot
| > mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional.
| > [    1.009844] Could not switch to high resolution mode on CPU 3
| > [    1.009848] Clockevents: could not switch to one-shot mode: lapic is not functional.
| > [    1.009852] Could not switch to high resolution mode on CPU 2
| > [    1.009855] Clockevents: could not switch to one-shot mode: lapic is not functional.
| > [    1.009858] Could not switch to high resolution mode on CPU 1
| > [    1.009969]  lapic is not functional.
| > [    1.056944] Could not switch to high resolution mode on CPU 0
| 
| No, that's documented behaviour:
| 
| > > > [    0.126660] APIC timer registered as dummy, due to nmi_watchdog=1!
| 
| Can you try nmi_watchdog=2 ?
| 
| Thanks,
| 
| 	tglx
| 

And get apic=debug a try too please. I remember there
was a problem with SB600 on ACPI side (but they should
be already fixed)

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-16 17:56             ` Cyrill Gorcunov
@ 2008-09-16 17:57               ` Cyrill Gorcunov
  0 siblings, 0 replies; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-16 17:57 UTC (permalink / raw)
  To: Thomas Gleixner, Joshua Hoblitt, Ingo Molnar, Andrew Morton,
	linux-kernel, bugme-daemon, j_kernel

[Cyrill Gorcunov - Tue, Sep 16, 2008 at 09:56:29PM +0400]
...
| 
| And get apic=debug a try too please. I remember there
| was a problem with SB600 on ACPI side (but they should
| be already fixed)
| 
| 		- Cyrill -

Sorry Thomas, I meant to send the message to Joshua

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
@ 2008-09-17  2:43 j_kernel
  2008-09-17  7:13 ` Cyrill Gorcunov
  0 siblings, 1 reply; 20+ messages in thread
From: j_kernel @ 2008-09-17  2:43 UTC (permalink / raw)
  To: Cyrill Gorcunov
  Cc: Thomas Gleixner, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

In-Reply-To: <20080916175629.GE7187@lenovo>
On Tue, Sep 16, 2008 at 09:56:29PM +0400, Cyrill Gorcunov wrote:
> [Thomas Gleixner - Tue, Sep 16, 2008 at 07:14:40AM -0700]
> | Can you try nmi_watchdog=2 ?
> | 
> | Thanks,
> | 
> | 	tglx
> | 
> 
> And get apic=debug a try too please. I remember there
> was a problem with SB600 on ACPI side (but they should
> be already fixed)

Attached is the updated dmesg from adding the nmi_watchdog=2 and acpi=debug boot params.

-J

--

[-- Attachment #2: ipp005.dmesg --]
[-- Type: text/plain, Size: 47690 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Linux version 2.6.27-rc5-22033-gd26acd9-dirty (root@ipp000) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #2 SMP Fri Sep 12 10:04:18 HST 2008
[    0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=2 show_msr=1 apic=debug console=tty0 console=ttyS0,115200n8
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
[    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000cff60000 (usable)
[    0.000000]  BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000230000000 (usable)
[    0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff
[    0.000000] init_memory_mapping
[    0.000000]  0000000000 - 00cfe00000 page 2M
[    0.000000]  00cfe00000 - 00cff60000 page 4k
[    0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000
[    0.000000] last_map_addr: cff60000 end: cff60000
[    0.000000] init_memory_mapping
[    0.000000]  0100000000 - 0230000000 page 2M
[    0.000000] kernel direct mapping tables up to 230000000 @ c000-16000
[    0.000000] last_map_addr: 230000000 end: 230000000
[    0.000000] RAMDISK: 37e0f000 - 37fef850
[    0.000000] DMI present.
[    0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD )
[    0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD       XSDT    6040000  LTP        0)
[    0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S      6040000 PTL_    F4240)
[    0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA      CK8  6040000 MSFT  3000000)
[    0.000000] ACPI: FACS CFF6AFC0, 0040
[    0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD    POWERNOW  6040000 AMD         1)
[    0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD    HAMMER    6040000 AMD         1)
[    0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
[    0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
[    0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD  HPETTBL   6040000  LTP        1)
[    0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD       APIC    6040000  LTP        0)
[    0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
[    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-d0000000
[    0.000000] SRAT: Node 0 PXM 0 100000000-130000000
[    0.000000] SRAT: Node 1 PXM 1 130000000-230000000
[    0.000000] NUMA: Allocated memnodemap from 11000 - 15680
[    0.000000] NUMA: Using 20 for the hash shift.
[    0.000000] Bootmem setup node 0 0000000000000000-0000000130000000
[    0.000000]   NODE_DATA [0000000000015680 - 000000000002067f]
[    0.000000]   bootmap [0000000000021000 -  0000000000046fff] pages 26
[    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
[    0.000000]   #2 [0000200000 - 0000f96910]    TEXT DATA BSS ==> [0000200000 - 0000f96910]
[    0.000000]   #3 [0037e0f000 - 0037fef850]          RAMDISK ==> [0037e0f000 - 0037fef850]
[    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
[    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE ==> [0000008000 - 000000c000]
[    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE ==> [000000c000 - 0000011000]
[    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP ==> [0000011000 - 0000015680]
[    0.000000] Bootmem setup node 1 0000000130000000-0000000230000000
[    0.000000]   NODE_DATA [0000000130000000 - 000000013000afff]
[    0.000000]   bootmap [000000013000b000 -  000000013002afff] pages 20
[    0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE
[    0.000000]   #2 [0000200000 - 0000f96910]    TEXT DATA BSS
[    0.000000]   #3 [0037e0f000 - 0037fef850]          RAMDISK
[    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved
[    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE
[    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE
[    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP
[    0.000000] Scan SMP from ffff880000000000 for 1024 bytes.
[    0.000000] Scan SMP from ffff88000009fc00 for 1024 bytes.
[    0.000000] Scan SMP from ffff8800000f0000 for 65536 bytes.
[    0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600
[    0.000000]  [ffffe20000000000-ffffe200071fffff] PMD -> [ffff880028200000-ffff88002e1fffff] on node 0
[    0.000000]  [ffffe20007200000-ffffe2000d1fffff] PMD -> [ffff880130200000-ffff8801361fffff] on node 1
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00230000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x0000009d
[    0.000000]     0: 0x00000100 -> 0x000cff60
[    0.000000]     0: 0x00100000 -> 0x00130000
[    0.000000]     1: 0x00130000 -> 0x00230000
[    0.000000] On node 0 totalpages: 1048317
[    0.000000]   DMA zone: 307 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 823232 pages, LIFO batch:31
[    0.000000]   Normal zone: 192000 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 1048576
[    0.000000]   Normal zone: 1024000 pages, LIFO batch:31
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x1008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Setting APIC routing to flat
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] mapped APIC to ffffffffff5fc000 (        fee00000)
[    0.000000] mapped IOAPIC to ffffffffff5fb000 (00000000fec00000)
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ce000
[    0.000000] PM: Registered nosave memory: 00000000000ce000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000cff60000 - 00000000cff6a000
[    0.000000] PM: Registered nosave memory: 00000000cff6a000 - 00000000cff80000
[    0.000000] PM: Registered nosave memory: 00000000cff80000 - 00000000d0000000
[    0.000000] PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000
[    0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
[    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
[    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000
[    0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fff80000
[    0.000000] PM: Registered nosave memory: 00000000fff80000 - 0000000100000000
[    0.000000] Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
[    0.000000] PERCPU: Allocating 51872 bytes of per cpu data
[    0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 2039539
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc nmi_watchdog=2 show_msr=1 apic=debug console=tty0 console=ttyS0,115200n8
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[    0.000000] Extended CMOS year: 2000
[    0.000000] TSC: PIT calibration confirmed by PMTIMER.
[    0.000000] TSC: using PIT calibration value
[    0.000000] Detected 2799.937 MHz processor.
[    0.003333] spurious 8259A interrupt: IRQ7.
[    0.003333] Console: colour VGA+ 80x25
[    0.003333] console [tty0] enabled
[    0.003333] console [ttyS0] enabled
[    0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.003333] ... MAX_LOCKDEP_SUBCLASSES:    8
[    0.003333] ... MAX_LOCK_DEPTH:          48
[    0.003333] ... MAX_LOCKDEP_KEYS:        8191
[    0.003333] ... CLASSHASH_SIZE:           4096
[    0.003333] ... MAX_LOCKDEP_ENTRIES:     8192
[    0.003333] ... MAX_LOCKDEP_CHAINS:      16384
[    0.003333] ... CHAINHASH_SIZE:          8192
[    0.003333]  memory used by lock dependency info: 3839 kB
[    0.003333]  per task-struct memory footprint: 1920 bytes
[    0.003333] Checking aperture...
[    0.003333] No AGP bridge found
[    0.003333] Node 0: aperture @ 0 size 32 MB
[    0.003333] Your BIOS doesn't leave a aperture memory hole
[    0.003333] Please enable the IOMMU option in the BIOS setup
[    0.003333] This costs you 64 MB of RAM
[    0.003333] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.003333] PM: Registered nosave memory: 0000000020000000 - 0000000024000000
[    0.003333] Memory: 8108304k/9175040k available (3897k kernel code, 279268k reserved, 2242k data, 676k init)
[    0.003333] CPA: page pool initialized 1 of 1 pages preallocated
[    0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.003333] hpet clockevent registered
[    0.003333] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.20 BogoMIPS (lpj=9333123)
[    0.006694] Security Framework initialized
[    0.010741] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.019180] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.025703] Mount-cache hash table entries: 256
[    0.026977] Initializing cgroup subsys ns
[    0.030005] Initializing cgroup subsys cpuacct
[    0.033335] Initializing cgroup subsys memory
[    0.036674] Initializing cgroup subsys devices
[    0.040006] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.043332] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.046666] CPU 0/0 -> Node 0
[    0.049999] tseg: 00cff80000
[    0.050000] CPU: Physical Processor ID: 0
[    0.053331] CPU: Processor Core ID: 0
[    0.056670] using C1E aware idle routine
[    0.060018] ACPI: Core revision 20080609
[    0.070057] Getting VERSION: 80050010
[    0.073331] Getting VERSION: 80050010
[    0.076663] Getting ID: 0
[    0.079466] Getting ID: ff000000
[    0.079994] Getting LVT0: 10000
[    0.079997] Getting LVT1: 10000
[    0.083331] masked ExtINT on CPU#0
[    0.086902] ENABLING IO-APIC IRQs
[    0.089996] init IO_APIC IRQs
[    0.089997]  IO-APIC (apicid-pin) 4-0 not connected.
[    0.090004] IOAPIC[0]: Set routing entry (4-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
[    0.090011] IOAPIC[0]: Set routing entry (4-2 -> 0x30 -> IRQ 0 Mode:0 Active:0)
[    0.090018] IOAPIC[0]: Set routing entry (4-3 -> 0x33 -> IRQ 3 Mode:0 Active:0)
[    0.090024] IOAPIC[0]: Set routing entry (4-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
[    0.090031] IOAPIC[0]: Set routing entry (4-5 -> 0x35 -> IRQ 5 Mode:0 Active:0)
[    0.090037] IOAPIC[0]: Set routing entry (4-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
[    0.090044] IOAPIC[0]: Set routing entry (4-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
[    0.090050] IOAPIC[0]: Set routing entry (4-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
[    0.090057] IOAPIC[0]: Set routing entry (4-9 -> 0x39 -> IRQ 9 Mode:1 Active:1)
[    0.090064] IOAPIC[0]: Set routing entry (4-10 -> 0x3a -> IRQ 10 Mode:0 Active:0)
[    0.090070] IOAPIC[0]: Set routing entry (4-11 -> 0x3b -> IRQ 11 Mode:0 Active:0)
[    0.090077] IOAPIC[0]: Set routing entry (4-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
[    0.090083] IOAPIC[0]: Set routing entry (4-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
[    0.090090] IOAPIC[0]: Set routing entry (4-14 -> 0x3e -> IRQ 14 Mode:0 Active:0)
[    0.090096] IOAPIC[0]: Set routing entry (4-15 -> 0x3f -> IRQ 15 Mode:0 Active:0)
[    0.090102]  IO-APIC (apicid-pin) 4-16, 4-17, 4-18, 4-19, 4-20, 4-21, 4-22, 4-23 not connected.
[    0.090269] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.128401] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.134745] Using local APIC timer interrupts.
[    0.139992] APIC timer calibration result 12499723
[    0.139993] Detected 12.499 MHz APIC timer.
[    0.143480] lockdep: fixing up alternatives.
[    0.146682] Booting processor 1/1 ip 6000
[    0.003333] Initializing CPU#1
[    0.003333] masked ExtINT on CPU#1
[    0.003333] Calibrating delay using timer specific routine.. 5602.22 BogoMIPS (lpj=9333163)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 1/1 -> Node 0
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
[    0.243715] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.253369] lockdep: fixing up alternatives.
[    0.256664] Booting processor 2/2 ip 6000
[    0.003333] Initializing CPU#2
[    0.003333] masked ExtINT on CPU#2
[    0.003333] Calibrating delay using timer specific routine.. 5602.19 BogoMIPS (lpj=9333118)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 2/2 -> Node 1
[    0.003333] CPU: Physical Processor ID: 1
[    0.003333] CPU: Processor Core ID: 0
[    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 0x7010600070106
[    0.353887] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.363387] lockdep: fixing up alternatives.
[    0.366664] Booting processor 3/3 ip 6000
[    0.003333] Initializing CPU#3
[    0.003333] masked ExtINT on CPU#3
[    0.003333] Calibrating delay using timer specific routine.. 5602.23 BogoMIPS (lpj=9333181)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 3/3 -> Node 1
[    0.003333] CPU: Physical Processor ID: 1
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106
[    0.463879] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.473316] Brought up 4 CPUs
[    0.476465] Total of 4 processors activated (22408.86 BogoMIPS).
[    0.476698] Testing NMI watchdog ... OK.
[    0.550266] net_namespace: 968 bytes
[    0.553490] xor: automatically using best checksumming function: generic_sse
[    0.573302]    generic_sse:  8630.400 MB/sec
[    0.576635] xor: using function: generic_sse (8630.400 MB/sec)
[    0.580028] NET: Registered protocol family 16
[    0.583586] No dock devices found.
[    0.586837] node 0 link 2: io port [1000, 3fff]
[    0.586839] TOM: 00000000d0000000 aka 3328M
[    0.589969] node 0 link 2: mmio [d0000000, dfffffff]
[    0.589972] node 0 link 2: mmio [a0000, bffff]
[    0.589974] node 0 link 2: mmio [e0000000, e7ffffff]
[    0.589976] TOM2: 0000000230000000 aka 8960M
[    0.593302] bus: [00,ff] on node 0 link 2
[    0.593304] bus: 00 index 0 io port: [0, ffff]
[    0.593306] bus: 00 index 1 mmio: [d0000000, ffffffff]
[    0.593307] bus: 00 index 2 mmio: [a0000, bffff]
[    0.593309] bus: 00 index 3 mmio: [230000000, fcffffffff]
[    0.593324] ACPI: bus type pci registered
[    0.596780] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4
[    0.599969] PCI: MCFG area at e0000000 reserved in E820
[    0.603407] PCI: Using MMCONFIG at e0000000 - e04fffff
[    0.606635] PCI: Using configuration type 1 for base access
[    0.617553] ACPI: EC: Look up EC in DSDT
[    0.621933] ACPI: Interpreter enabled
[    0.623302] ACPI: (supports S0 S1 S3 S4 S5)
[    0.630429] ACPI: Using IOAPIC for interrupt routing
[    0.643218] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    0.643602] pci 0000:00:01.1: PME# supported from D3hot D3cold
[    0.646639] pci 0000:00:01.1: PME# disabled
[    0.650008] pci 0000:00:02.0: supports D1
[    0.650009] pci 0000:00:02.0: supports D2
[    0.650011] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.653303] pci 0000:00:02.0: PME# disabled
[    0.656671] pci 0000:00:02.1: supports D1
[    0.656673] pci 0000:00:02.1: supports D2
[    0.656674] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold
[    0.659969] pci 0000:00:02.1: PME# disabled
[    0.663528] pci 0000:00:08.0: supports D1
[    0.663530] pci 0000:00:08.0: supports D2
[    0.663531] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.666637] pci 0000:00:08.0: PME# disabled
[    0.670016] pci 0000:00:09.0: supports D1
[    0.670017] pci 0000:00:09.0: supports D2
[    0.670019] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.673304] pci 0000:00:09.0: PME# disabled
[    0.676666] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.679969] pci 0000:00:0a.0: PME# disabled
[    0.683330] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.686636] pci 0000:00:0d.0: PME# disabled
[    0.689998] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.693302] pci 0000:00:0f.0: PME# disabled
[    0.696873] pci 0000:01:04.0: supports D1
[    0.696875] pci 0000:01:04.0: supports D2
[    0.696898] pci 0000:00:06.0: transparent bridge
[    0.699969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff]
[    0.703303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff]
[    0.706636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff]
[    0.710035] pci 0000:02:00.0: supports D1
[    0.710066] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff]
[    0.713386] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.713458] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT]
[    0.713481] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT]
[    0.713508] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT]
[    0.713533] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT]
[    0.739522] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.746786] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.756769] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.767155] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.778473] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.789236] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.795676] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.803377] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.813825] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.825139] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.834586] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.844277] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 18 19 20 21 22 23)
[    0.853963] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.863825] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.873830] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.885589] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.891879] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.901564] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.911393] Linux Plug and Play Support v0.97 (c) Adam Belay
[    0.913365] pnp: PnP ACPI init
[    0.916646] ACPI: bus type pnp registered
[    0.920659] IOAPIC[0]: Set routing entry (4-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
[    0.921289] IOAPIC[0]: Set routing entry (4-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
[    0.921379] IOAPIC[0]: Set routing entry (4-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
[    0.923450] IOAPIC[0]: Set routing entry (4-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
[    0.923886] IOAPIC[0]: Set routing entry (4-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
[    0.924783] IOAPIC[0]: Set routing entry (4-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
[    0.925590] pnp: PnP ACPI: found 14 devices
[    0.926636] ACPI: ACPI bus type pnp unregistered
[    0.930238] SCSI subsystem initialized
[    0.933467] libata version 3.00 loaded.
[    0.933747] usbcore: registered new interface driver usbfs
[    0.936702] usbcore: registered new interface driver hub
[    0.940064] usbcore: registered new device driver usb
[    0.943673] PCI: Using ACPI for IRQ routing
[    0.946637] number of MP IRQ sources: 16.
[    0.946639] number of IO-APIC #4 registers: 24.
[    0.946640] testing the IO APIC.......................
[    0.949973] 
[    0.953302] IO APIC #4......
[    0.953303] .... register #00: 04000000
[    0.953304] .......    : physical APIC id: 04
[    0.953306] .... register #01: 00170011
[    0.953307] .......     : max redirection entries: 0017
[    0.953308] .......     : PRQ implemented: 0
[    0.953310] .......     : IO APIC version: 0011
[    0.953311] .... register #02: 04000000
[    0.953312] .......     : arbitration: 04
[    0.953314] .... IRQ redirection table:
[    0.953315]  NR Dst Mask Trig IRR Pol Stat Dmod Deli Vect:   
[    0.953320]  00 000 1    0    0   0   0    0    0    00
[    0.953326]  01 00F 0    0    0   0   0    1    1    31
[    0.953331]  02 00F 0    0    0   0   0    1    1    30
[    0.953337]  03 00F 0    0    0   0   0    1    1    33
[    0.953343]  04 00F 0    0    0   0   0    1    1    34
[    0.953348]  05 00F 1    0    0   0   0    1    1    35
[    0.953354]  06 00F 0    0    0   0   0    1    1    36
[    0.953360]  07 00F 0    0    0   0   0    1    1    37
[    0.953365]  08 00F 0    0    0   0   0    1    1    38
[    0.953371]  09 00F 0    1    0   1   0    1    1    39
[    0.953377]  0a 00F 1    0    0   0   0    1    1    3A
[    0.953382]  0b 00F 1    0    0   0   0    1    1    3B
[    0.953388]  0c 00F 0    0    0   0   0    1    1    3C
[    0.953393]  0d 00F 0    0    0   0   0    1    1    3D
[    0.953399]  0e 00F 0    0    0   0   0    1    1    3E
[    0.953405]  0f 00F 0    0    0   0   0    1    1    3F
[    0.953410]  10 000 1    0    0   0   0    0    0    00
[    0.953416]  11 000 1    0    0   0   0    0    0    00
[    0.953422]  12 000 1    0    0   0   0    0    0    00
[    0.953427]  13 000 1    0    0   0   0    0    0    00
[    0.953433]  14 000 1    0    0   0   0    0    0    00
[    0.953438]  15 000 1    0    0   0   0    0    0    00
[    0.953444]  16 000 1    0    0   0   0    0    0    00
[    0.953450]  17 000 1    0    0   0   0    0    0    00
[    0.953452] IRQ to pin mappings:
[    0.953453] IRQ0 -> 0:2
[    0.953455] IRQ1 -> 0:1
[    0.953457] IRQ3 -> 0:3
[    0.953459] IRQ4 -> 0:4
[    0.953460] IRQ5 -> 0:5
[    0.953462] IRQ6 -> 0:6
[    0.953464] IRQ7 -> 0:7
[    0.953466] IRQ8 -> 0:8
[    0.953467] IRQ9 -> 0:9
[    0.953469] IRQ10 -> 0:10
[    0.953471] IRQ11 -> 0:11
[    0.953472] IRQ12 -> 0:12
[    0.953474] IRQ13 -> 0:13
[    0.953476] IRQ14 -> 0:14
[    0.953478] IRQ15 -> 0:15
[    0.953483] .................................... done.
[    0.973557] PCI-DMA: Disabling AGP.
[    0.977781] PCI-DMA: aperture base @ 20000000 size 65536 KB
[    0.979968] PCI-DMA: using GART IOMMU.
[    0.983304] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    0.987206] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
[    0.993677] hpet0: 3 32-bit timers, 25000000 Hz
[    0.997730] ACPI: RTC can wake from S4
[    0.999979] Switched to high resolution mode on CPU 0
[    1.000026] Switched to high resolution mode on CPU 3
[    1.000047] Switched to high resolution mode on CPU 2
[    1.003246] Switched to high resolution mode on CPU 1
[    1.013368] system 00:00: iomem range 0xffc00000-0xffffffff could not be reserved
[    1.021203] system 00:00: iomem range 0xfec00000-0xfec00fff could not be reserved
[    1.029036] system 00:00: iomem range 0xfee00000-0xfeefffff could not be reserved
[    1.036872] system 00:00: iomem range 0xfed00000-0xfed00fff has been reserved
[    1.044213] system 00:03: iomem range 0xd0000000-0xd0007fff has been reserved
[    1.051552] system 00:04: ioport range 0x1000-0x107f has been reserved
[    1.058279] system 00:04: ioport range 0x1080-0x10ff has been reserved
[    1.065003] system 00:04: ioport range 0x1400-0x147f has been reserved
[    1.071729] system 00:04: ioport range 0x1480-0x14ff has been reserved
[    1.078456] system 00:04: ioport range 0x1800-0x187f has been reserved
[    1.085183] system 00:04: ioport range 0x1880-0x18ff has been reserved
[    1.091907] system 00:04: ioport range 0x2440-0x247f has been reserved
[    1.098632] system 00:04: ioport range 0x2400-0x243f has been reserved
[    1.105366] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
[    1.111916] system 00:07: ioport range 0xc05-0xc06 has been reserved
[    1.119679] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01
[    1.125800] pci 0000:00:06.0:   IO window: 0x3000-0x3fff
[    1.131311] pci 0000:00:06.0:   MEM window: 0xd0100000-0xd01fffff
[    1.137610] pci 0000:00:06.0:   PREFETCH window: 0x000000d8000000-0x000000dfffffff
[    1.145526] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02
[    1.151643] pci 0000:00:0a.0:   IO window: disabled
[    1.156719] pci 0000:00:0a.0:   MEM window: 0xd0200000-0xd02fffff
[    1.163006] pci 0000:00:0a.0:   PREFETCH window: 0x000000d1000000-0x000000d10fffff
[    1.170932] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03
[    1.177044] pci 0000:00:0d.0:   IO window: disabled
[    1.182114] pci 0000:00:0d.0:   MEM window: disabled
[    1.187271] pci 0000:00:0d.0:   PREFETCH window: disabled
[    1.192861] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04
[    1.198979] pci 0000:00:0f.0:   IO window: disabled
[    1.204048] pci 0000:00:0f.0:   MEM window: disabled
[    1.209205] pci 0000:00:0f.0:   PREFETCH window: disabled
[    1.214801] pci 0000:00:06.0: setting latency timer to 64
[    1.214807] pci 0000:00:0a.0: setting latency timer to 64
[    1.214812] pci 0000:00:0d.0: setting latency timer to 64
[    1.214817] pci 0000:00:0f.0: setting latency timer to 64
[    1.214820] bus: 00 index 0 io port: [0, ffff]
[    1.219456] bus: 00 index 1 mmio: [0, ffffffffffffffff]
[    1.224874] bus: 01 index 0 io port: [3000, 3fff]
[    1.229771] bus: 01 index 1 mmio: [d0100000, d01fffff]
[    1.239972] bus: 01 index 2 mmio: [d8000000, dfffffff]
[    1.245301] bus: 01 index 3 io port: [0, ffff]
[    1.249937] bus: 01 index 4 mmio: [0, ffffffffffffffff]
[    1.255354] bus: 02 index 0 mmio: [0, 0]
[    1.259472] bus: 02 index 1 mmio: [d0200000, d02fffff]
[    1.264801] bus: 02 index 2 mmio: [d1000000, d10fffff]
[    1.270132] bus: 02 index 3 mmio: [0, 0]
[    1.274253] bus: 03 index 0 mmio: [0, 0]
[    1.278370] bus: 03 index 1 mmio: [0, 0]
[    1.282488] bus: 03 index 2 mmio: [0, 0]
[    1.286605] bus: 03 index 3 mmio: [0, 0]
[    1.290722] bus: 04 index 0 mmio: [0, 0]
[    1.294838] bus: 04 index 1 mmio: [0, 0]
[    1.298955] bus: 04 index 2 mmio: [0, 0]
[    1.303073] bus: 04 index 3 mmio: [0, 0]
[    1.307207] NET: Registered protocol family 2
[    1.343606] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    1.353558] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    1.366101] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
[    1.375860] TCP: Hash tables configured (established 524288 bind 65536)
[    1.382707] TCP reno registered
[    1.393492] NET: Registered protocol family 1
[    1.398166] checking if image is initramfs... it is
[    1.501735] Freeing initrd memory: 1922k freed
[    1.507469] Simple Boot Flag at 0x37 set to 0x80
[    1.515235] audit: initializing netlink socket (disabled)
[    1.520899] type=2000 audit(1221619050.520:1): initialized
[    1.532146] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.542716] VFS: Disk quotas dquot_6.5.1
[    1.546980] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.555347] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    1.562281] msgmni has been set to 15840
[    1.566647] async_tx: api initialized (async)
[    1.571215] io scheduler noop registered
[    1.575345] io scheduler anticipatory registered
[    1.580169] io scheduler deadline registered
[    1.584841] io scheduler cfq registered (default)
[    1.589785] pci 0000:00:00.0: Enabling HT MSI Mapping
[    1.797529] pci 0000:00:05.0: Enabling HT MSI Mapping
[    1.802799] pci 0000:00:05.1: Enabling HT MSI Mapping
[    1.808074] pci 0000:00:05.2: Enabling HT MSI Mapping
[    1.813351] pci 0000:00:06.0: Enabling HT MSI Mapping
[    1.818632] pci 0000:00:08.0: Enabling HT MSI Mapping
[    1.823910] pci 0000:00:09.0: Enabling HT MSI Mapping
[    1.829182] pci 0000:00:0a.0: Enabling HT MSI Mapping
[    1.834452] pci 0000:00:0d.0: Enabling HT MSI Mapping
[    1.839721] pci 0000:00:0f.0: Enabling HT MSI Mapping
[    1.844991] pci 0000:01:04.0: Boot video device
[    1.845193] pcieport-driver 0000:00:0a.0: setting latency timer to 64
[    1.845222] pcieport-driver 0000:00:0a.0: found MSI capability
[    1.851316] pci_express 0000:00:0a.0:pcie00: allocate port service
[    1.851407] pci_express 0000:00:0a.0:pcie01: allocate port service
[    1.851468] pci_express 0000:00:0a.0:pcie03: allocate port service
[    1.851570] pcieport-driver 0000:00:0d.0: setting latency timer to 64
[    1.851596] pcieport-driver 0000:00:0d.0: found MSI capability
[    1.857661] pci_express 0000:00:0d.0:pcie00: allocate port service
[    1.857732] pci_express 0000:00:0d.0:pcie01: allocate port service
[    1.857791] pci_express 0000:00:0d.0:pcie03: allocate port service
[    1.857892] pcieport-driver 0000:00:0f.0: setting latency timer to 64
[    1.857923] pcieport-driver 0000:00:0f.0: found MSI capability
[    1.863984] pci_express 0000:00:0f.0:pcie00: allocate port service
[    1.864056] pci_express 0000:00:0f.0:pcie01: allocate port service
[    1.864116] pci_express 0000:00:0f.0:pcie03: allocate port service
[    1.865960] aer 0000:00:0a.0:pcie01: AER service couldn't init device: no _OSC support
[    1.867614] aer 0000:00:0d.0:pcie01: AER service couldn't init device: no _OSC support
[    1.869243] aer 0000:00:0f.0:pcie01: AER service couldn't init device: no _OSC support
[    1.869694] input: Power Button (FF) as /class/input/input0
[    1.875475] ACPI: Power Button (FF) [PWRF]
[    1.879888] input: Power Button (CM) as /class/input/input1
[    1.885657] ACPI: Power Button (CM) [PWRB]
[    1.890403] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[    1.896470] processor ACPI0007:00: registered as cooling_device0
[    1.902681] ACPI: Processor [C000] (supports 2 throttling states)
[    1.909166] processor ACPI0007:01: registered as cooling_device1
[    1.915478] processor ACPI0007:02: registered as cooling_device2
[    1.921779] processor ACPI0007:03: registered as cooling_device3
[    2.001837] hpet_resources: 0xfed00000 is busy
[    2.002038] Non-volatile memory driver v1.2
[    2.006568] Linux agpgart interface v0.103
[    2.010966] Serial: 8250/16550 driver4 ports, IRQ sharing disabled
[    2.017552] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.024529] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.030642] Floppy drive(s): fd0 is 1.44M
[    5.046687] floppy0: no floppy controllers found
[    5.054215] brd: module loaded
[    5.058709] loop: module loaded
[    5.062049] tun: Universal TUN/TAP device driver, 1.6
[    5.067295] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    5.073870] console [netcon0] enabled
[    5.077726] netconsole: network logging started
[    5.082451] Uniform Multi-Platform E-IDE driver
[    5.087308] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
[    5.097421] Probing IDE interface ide0...
[    6.063534] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive
[    6.120118] Probing IDE interface ide1...
[    6.653446] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[    6.658285] ide1 at 0x170-0x177,0x376 on irq 15
[    6.665085] hdb: ATAPI 24X CD-ROM drive, 256kB Cache
[    6.670460] Uniform CD-ROM driver Revision: 3.20
[    6.685431] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
[    6.693244] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[    6.700497] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008
[    6.706688] Driver 'sd' needs updating - please use bus_type methods
[    6.713309] Driver 'sr' needs updating - please use bus_type methods
[    6.720482] sata_nv 0000:00:05.0: version 3.5
[    6.720779] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23
[    6.726726] IOAPIC[0]: Set routing entry (4-23 -> 0x61 -> IRQ 23 Mode:1 Active:1)
[    6.726743] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 (level, low) -> IRQ 23
[    6.735353] sata_nv 0000:00:05.0: Using SWNCQ mode
[    6.740472] sata_nv 0000:00:05.0: setting latency timer to 64
[    6.740651] scsi0 : sata_nv
[    6.743839] scsi1 : sata_nv
[    6.747023] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq 23
[    6.754354] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq 23
[    7.090373] ata1: SATA link down (SStatus 0 SControl 300)
[    7.423705] ata2: SATA link down (SStatus 0 SControl 300)
[    7.429582] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22
[    7.435536] IOAPIC[0]: Set routing entry (4-22 -> 0x69 -> IRQ 22 Mode:1 Active:1)
[    7.435551] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 (level, low) -> IRQ 22
[    7.444165] sata_nv 0000:00:05.1: Using SWNCQ mode
[    7.449263] sata_nv 0000:00:05.1: setting latency timer to 64
[    7.449402] scsi2 : sata_nv
[    7.452526] scsi3 : sata_nv
[    7.455714] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq 22
[    7.463043] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq 22
[    7.797000] ata3: SATA link down (SStatus 0 SControl 300)
[    8.130331] ata4: SATA link down (SStatus 0 SControl 300)
[    8.136202] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21
[    8.142156] IOAPIC[0]: Set routing entry (4-21 -> 0x71 -> IRQ 21 Mode:1 Active:1)
[    8.142172] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 (level, low) -> IRQ 21
[    8.150785] sata_nv 0000:00:05.2: Using SWNCQ mode
[    8.155882] sata_nv 0000:00:05.2: setting latency timer to 64
[    8.156027] scsi4 : sata_nv
[    8.159160] scsi5 : sata_nv
[    8.162335] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq 21
[    8.169672] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq 21
[    8.507038] ata5: SATA link down (SStatus 0 SControl 300)
[    8.840372] ata6: SATA link down (SStatus 0 SControl 300)
[    8.846110] pata_amd 0000:00:04.0: version 0.3.10
[    8.846134] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region [0x1f0-0x1f7]
[    8.853891] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 (errno=-16)
[    8.862160] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region [0x170-0x177]
[    8.869906] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 (errno=-16)
[    8.878166] pata_amd 0000:00:04.0: no available native port
[    8.884290] Fusion MPT base driver 3.04.07
[    8.888582] Copyright (c) 1999-2008 LSI Corporation
[    8.893666] Fusion MPT SPI Host driver 3.04.07
[    8.898396] Fusion MPT SAS Host driver 3.04.07
[    8.903765] ieee1394: raw1394: /dev/raw1394 device initialized
[    8.910299] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20
[    8.916243] IOAPIC[0]: Set routing entry (4-20 -> 0x79 -> IRQ 20 Mode:1 Active:1)
[    8.916257] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 (level, low) -> IRQ 20
[    8.924965] ehci_hcd 0000:00:02.1: setting latency timer to 64
[    8.924968] ehci_hcd 0000:00:02.1: EHCI Host Controller
[    8.930641] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
[    8.938414] ehci_hcd 0000:00:02.1: debug port 1
[    8.943142] ehci_hcd 0000:00:02.1: cache line size of 64 is not supported
[    8.943159] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000
[    8.956681] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[    8.964675] usb usb1: configuration #1 chosen from 1 choice
[    8.970539] hub 1-0:1.0: USB hub found
[    8.974502] hub 1-0:1.0: 10 ports detected
[    9.080399] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
[    9.080655] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19
[    9.086607] IOAPIC[0]: Set routing entry (4-19 -> 0x81 -> IRQ 19 Mode:1 Active:1)
[    9.086622] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 (level, low) -> IRQ 19
[    9.095328] ohci_hcd 0000:00:02.0: setting latency timer to 64
[    9.095330] ohci_hcd 0000:00:02.0: OHCI Host Controller
[    9.100926] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
[    9.108695] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000
[    9.168822] usb usb2: configuration #1 chosen from 1 choice
[    9.174679] hub 2-0:1.0: USB hub found
[    9.178639] hub 2-0:1.0: 10 ports detected
[    9.283663] USB Universal Host Controller Interface driver v3.0
[    9.289954] usbcore: registered new interface driver usblp
[    9.295643] Initializing USB Mass Storage driver...
[    9.300787] usbcore: registered new interface driver usb-storage
[    9.306995] USB Mass Storage support registered.
[    9.311965] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12
[    9.570302] serio: i8042 KBD port at 0x60,0x64 irq 1
[    9.575666] mice: PS/2 mouse device common for all mice
[    9.613337] md: linear personality registered for level -1
[    9.780862] md: raid0 personality registered for level 0
[    9.786372] md: raid1 personality registered for level 1
[    9.791884] md: raid10 personality registered for level 10
[    9.853774] raid6: int64x1   2557 MB/s
[    9.913770] raid6: int64x2   3277 MB/s
[    9.973759] raid6: int64x4   3227 MB/s
[   10.033775] raid6: int64x8   2269 MB/s
[   10.093762] raid6: sse2x1    3908 MB/s
[   10.153763] raid6: sse2x2    5114 MB/s
[   10.213758] raid6: sse2x4    5239 MB/s
[   10.217700] raid6: using algorithm sse2x4 (5239 MB/s)
[   10.222948] md: raid6 personality registered for level 6
[   10.228459] md: raid5 personality registered for level 5
[   10.233971] md: raid4 personality registered for level 4
[   10.244367] md: multipath personality registered for level -4
[   10.250567] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
[   10.259494] cpuidle: using governor ladder
[   10.264025] cpuidle: using governor menu
[   10.268217] usbcore: registered new interface driver usbhid
[   10.273985] usbhid: v2.6:USB HID core driver
[   10.278568] oprofile: using NMI interrupt.
[   10.283357] TCP cubic registered
[   10.287003] NET: Registered protocol family 10
[   10.292656] IPv6 over IPv4 tunneling driver
[   10.297796] NET: Registered protocol family 17
[   10.302830] RPC: Registered udp transport module.
[   10.307731] RPC: Registered tcp transport module.
[   10.312719] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 processors (4 cpu cores) (version 2.20.00)
[   10.316673] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
[   10.329561] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
[   10.335416] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
[   10.341275] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
[   10.347059] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
[   10.352912] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
[   10.358772] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
[   10.317508] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
[   10.370709] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
[   10.376570] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
[   10.382429] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
[   10.388200] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
[   10.394060] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
[   10.399918] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
[   10.406608] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[   10.413157] Freeing unused kernel memory: 676k freed
[   10.418705] Write protecting the kernel read-only data: 5672k
[   10.853151] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18
[   10.853161] IOAPIC[0]: Set routing entry (4-18 -> 0x89 -> IRQ 18 Mode:1 Active:1)
[   10.853223] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 (level, low) -> IRQ 18
[   10.853230] arcmsr 0000:02:00.0: setting latency timer to 64
[   10.866697] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31  
[   10.880018] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 capable)
[   10.880019]  Driver Version 1.20.00.15 2008/02/27
[   10.881528] scsi 6:0:0:0: Direct-Access     Areca    ARC-1280-VOL#00  R001 PQ: 0 ANSI: 5
[   10.882676] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[   10.882721] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
[   10.882768] sd 6:0:0:0: [sda] Write Protect is off
[   10.882772] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
[   10.882851] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   10.883023] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[   10.883067] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
[   10.883108] sd 6:0:0:0: [sda] Write Protect is off
[   10.883111] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
[   10.883190] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   10.883195]  sda: sda1 sda2 sda3 sda4
[   10.915493] sd 6:0:0:0: [sda] Attached SCSI disk
[   10.916329] sd 6:0:0:0: Attached scsi generic sg0 type 0
[   10.917710] scsi 6:0:16:0: Processor         Areca    RAID controller  R001 PQ: 0 ANSI: 0
[   10.918870] scsi 6:0:16:0: Attached scsi generic sg1 type 3
[   11.000429] Clocksource tsc unstable (delta = -89318010 ns)
[   11.073271] 3ware Storage Controller device driver for Linux v1.26.02.002.
[   11.094544] 3ware 9000 Storage Controller device driver for Linux v2.26.02.011.
[   11.128602] Adaptec aacraid driver 1.1-5[2456]-ms
[   11.402730] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
[   11.408669] SGI XFS Quota Management subsystem
[   11.459380] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
[   11.459388] Copyright (c) 1999-2006 Intel Corporation.
[   14.751255] kjournald starting.  Commit interval 5 seconds
[   14.751255] EXT3-fs: mounted filesystem with ordered data mode.
[   17.062543] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2440
[   17.062600] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2400
[   17.073911] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.61.
[   17.074765] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 17
[   17.074780] IOAPIC[0]: Set routing entry (4-17 -> 0x91 -> IRQ 17 Mode:1 Active:1)
[   17.074895] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 17 (level, low) -> IRQ 17
[   17.074908] forcedeth 0000:00:08.0: setting latency timer to 64
[   17.079543] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 0, addr 00:e0:81:76:28:d6
[   17.079558] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
[   17.080351] ACPI: PCI Interrupt Link [LMA2] enabled at IRQ 16
[   17.080363] IOAPIC[0]: Set routing entry (4-16 -> 0x99 -> IRQ 16 Mode:1 Active:1)
[   17.080418] forcedeth 0000:00:09.0: PCI INT A -> Link[LMA2] -> GSI 16 (level, low) -> IRQ 16
[   17.080427] forcedeth 0000:00:09.0: setting latency timer to 64
[   17.083446] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 1, addr 00:e0:81:76:28:d7
[   17.083454] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt timirq gbit lnktim msi desc-v3
[   17.972394] EXT3 FS on sda3, internal journal
[   18.304174] SMsC 37B787 watchdog component driver 1.1 initialising...
[   18.305555] smsc37b787_wdt: Timeout set to 60 second(s).
[   18.305560] smsc37b787_wdt: Watchdog initialized and sleeping (nowayout=0)...
[   20.567087] kjournald starting.  Commit interval 5 seconds
[   20.590125] EXT3 FS on sda4, internal journal
[   20.590141] EXT3-fs: mounted filesystem with journal data mode.
[   20.644881] Adding 15999992k swap on /dev/sda2.  Priority:-1 extents:1 across:15999992k
[   40.560439] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   40.567835] NFSD: starting 90-second grace period
[   42.150031] eth0: no IPv6 routers present
[   48.719390] w83627hf: Found W83627HF chip at 0xc00
[   49.799833] kjournald starting.  Commit interval 5 seconds
[   49.800228] EXT3 FS on sda1, internal journal
[   49.800255] EXT3-fs: mounted filesystem with ordered data mode.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17  2:43 [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ??? j_kernel
@ 2008-09-17  7:13 ` Cyrill Gorcunov
  2008-09-17  9:20   ` j_kernel
  0 siblings, 1 reply; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-17  7:13 UTC (permalink / raw)
  To: j_kernel
  Cc: Thomas Gleixner, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon

[j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 04:43:12PM -1000]
| In-Reply-To: <20080916175629.GE7187@lenovo>
| On Tue, Sep 16, 2008 at 09:56:29PM +0400, Cyrill Gorcunov wrote:
| > [Thomas Gleixner - Tue, Sep 16, 2008 at 07:14:40AM -0700]
| > | Can you try nmi_watchdog=2 ?
| > | 
| > | Thanks,
| > | 
| > | 	tglx
| > | 
| > 
| > And get apic=debug a try too please. I remember there
| > was a problem with SB600 on ACPI side (but they should
| > be already fixed)
| 
| Attached is the updated dmesg from adding the nmi_watchdog=2 and acpi=debug boot params.
| 
| -J
| 

Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)?
Not sure if it help but could be usefull info. Thanks for this dmesg - trying
to analize it.

(
here is how to do it
	http://kernel.org/pub/linux/kernel/people/helgaas/debug
	http://lwn.net/Articles/237085/
)

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17  7:13 ` Cyrill Gorcunov
@ 2008-09-17  9:20   ` j_kernel
  2008-09-17  9:38     ` Cyrill Gorcunov
                       ` (2 more replies)
  0 siblings, 3 replies; 20+ messages in thread
From: j_kernel @ 2008-09-17  9:20 UTC (permalink / raw)
  To: Cyrill Gorcunov
  Cc: Thomas Gleixner, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon

On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote:
> Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)?

Done but I was unable to extract the DSDT table as described in the
instructions.

#  acpixtract DSDT < acpidump.asc           
Could not open DSDT

Cheers,

-J

--

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17  9:20   ` j_kernel
@ 2008-09-17  9:38     ` Cyrill Gorcunov
  2008-09-17 12:59     ` Cyrill Gorcunov
  2008-09-17 16:27     ` Cyrill Gorcunov
  2 siblings, 0 replies; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-17  9:38 UTC (permalink / raw)
  To: j_kernel
  Cc: Thomas Gleixner, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon

[j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 11:20:58PM -1000]
| On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote:
| > Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)?
| 
| Done but I was unable to extract the DSDT table as described in the
| instructions.
| 
| #  acpixtract DSDT < acpidump.asc           
| Could not open DSDT
| 
| Cheers,
| 
| -J
| 

 it's enough, thanks!

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17  9:20   ` j_kernel
  2008-09-17  9:38     ` Cyrill Gorcunov
@ 2008-09-17 12:59     ` Cyrill Gorcunov
  2008-09-17 13:13       ` Ingo Molnar
  2008-09-17 16:27     ` Cyrill Gorcunov
  2 siblings, 1 reply; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-17 12:59 UTC (permalink / raw)
  To: j_kernel
  Cc: Thomas Gleixner, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon

[j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 11:20:58PM -1000]
| On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote:
| > Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)?
| 
| Done but I was unable to extract the DSDT table as described in the
| instructions.
| 
| #  acpixtract DSDT < acpidump.asc           
| Could not open DSDT
| 
| Cheers,
| 
| -J
| 
| --
| 

As far as I can see there is no really issue in ACPI configuration,
at least on apic side...

Evetually nmi_watchdog=2 does work right? Though it's not
good that we've got 8259 spurious interrupt. Hmm...

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17 12:59     ` Cyrill Gorcunov
@ 2008-09-17 13:13       ` Ingo Molnar
  2008-09-17 13:29         ` Cyrill Gorcunov
  0 siblings, 1 reply; 20+ messages in thread
From: Ingo Molnar @ 2008-09-17 13:13 UTC (permalink / raw)
  To: Cyrill Gorcunov
  Cc: j_kernel, Thomas Gleixner, Andrew Morton, linux-kernel,
	bugme-daemon


> Evetually nmi_watchdog=2 does work right? Though it's not good that 
> we've got 8259 spurious interrupt. Hmm...

well, nmi_watchdog=2 changes the layout of clockevent devices and easily 
switches the system into non-highres non-dynticks mode. So it can hide 
bugs.

	Ingo

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17 13:13       ` Ingo Molnar
@ 2008-09-17 13:29         ` Cyrill Gorcunov
  0 siblings, 0 replies; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-17 13:29 UTC (permalink / raw)
  To: Ingo Molnar
  Cc: j_kernel, Thomas Gleixner, Andrew Morton, linux-kernel,
	bugme-daemon

[Ingo Molnar - Wed, Sep 17, 2008 at 03:13:06PM +0200]
| 
| > Evetually nmi_watchdog=2 does work right? Though it's not good that 
| > we've got 8259 spurious interrupt. Hmm...
| 
| well, nmi_watchdog=2 changes the layout of clockevent devices and easily 
| switches the system into non-highres non-dynticks mode. So it can hide 
| bugs.
| 
| 	Ingo
| 

and it does since we have nmi stuck on lapics. Still investigating.

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17  9:20   ` j_kernel
  2008-09-17  9:38     ` Cyrill Gorcunov
  2008-09-17 12:59     ` Cyrill Gorcunov
@ 2008-09-17 16:27     ` Cyrill Gorcunov
  2008-09-17 20:48       ` j_kernel
  2 siblings, 1 reply; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-17 16:27 UTC (permalink / raw)
  To: j_kernel
  Cc: Thomas Gleixner, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon

[j_kernel@hoblitt.com - Tue, Sep 16, 2008 at 11:20:58PM -1000]
| On Wed, Sep 17, 2008 at 11:13:57AM +0400, Cyrill Gorcunov wrote:
| > Joshua, could you please attach ACPI tables dump to the bugzilla entry (~300K)?
| 
| Done but I was unable to extract the DSDT table as described in the
| instructions.
| 
| #  acpixtract DSDT < acpidump.asc           
| Could not open DSDT
| 
| Cheers,
| 
| -J
| 
| --
| 

Joshua, could you please do one more thing - boot the kernel
with "debug apic=debug nmi_watchdog=1" (among others) and
publish dmsg?

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17 16:27     ` Cyrill Gorcunov
@ 2008-09-17 20:48       ` j_kernel
  2008-09-18  7:09         ` Cyrill Gorcunov
  0 siblings, 1 reply; 20+ messages in thread
From: j_kernel @ 2008-09-17 20:48 UTC (permalink / raw)
  To: Cyrill Gorcunov, Ingo Molnar, Andrew Morton, linux-kernel,
	bugme-daemon
  Cc: Thomas Gleixner

[-- Attachment #1: Type: text/plain, Size: 315 bytes --]

On Wed, Sep 17, 2008 at 08:27:04PM +0400, Cyrill Gorcunov wrote:
> Joshua, could you please do one more thing - boot the kernel
> with "debug apic=debug nmi_watchdog=1" (among others) and
> publish dmsg?

dmesg from "debug apic=debug nmi_watchdog=2" is attached.
                                          ^

-J

--

[-- Attachment #2: ipp005-debug.dmesg --]
[-- Type: text/plain, Size: 45391 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Linux version 2.6.27-rc5-22033-gd26acd9-dirty (root@ipp000) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #2 SMP Fri Sep 12 10:04:18 HST 2008
[    0.000000] Command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc debug nmi_watchdog=2 show_msr=1 apic=debug console=tty0 console=ttyS0,115200n8
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
[    0.000000]  BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000ce000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000cff60000 (usable)
[    0.000000]  BIOS-e820: 00000000cff60000 - 00000000cff6a000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000cff6a000 - 00000000cff80000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000cff80000 - 00000000d0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000230000000 (usable)
[    0.000000] last_pfn = 0x230000 max_arch_pfn = 0x3ffffffff
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xcff60 max_arch_pfn = 0x3ffffffff
[    0.000000] init_memory_mapping
[    0.000000]  0000000000 - 00cfe00000 page 2M
[    0.000000]  00cfe00000 - 00cff60000 page 4k
[    0.000000] kernel direct mapping tables up to cff60000 @ 8000-e000
[    0.000000] last_map_addr: cff60000 end: cff60000
[    0.000000] init_memory_mapping
[    0.000000]  0100000000 - 0230000000 page 2M
[    0.000000] kernel direct mapping tables up to 230000000 @ c000-16000
[    0.000000] last_map_addr: 230000000 end: 230000000
[    0.000000] RAMDISK: 37e0f000 - 37fef850
[    0.000000] DMI present.
[    0.000000] ACPI: RSDP 000F75D0, 0024 (r2 PTLTD )
[    0.000000] ACPI: XSDT CFF656FA, 0064 (r1 PTLTD       XSDT    6040000  LTP        0)
[    0.000000] ACPI: FACP CFF6970E, 00F4 (r3 NVIDIA CK8S      6040000 PTL_    F4240)
[    0.000000] ACPI: DSDT CFF6575E, 3F3C (r1 NVIDIA      CK8  6040000 MSFT  3000000)
[    0.000000] ACPI: FACS CFF6AFC0, 0040
[    0.000000] ACPI: SSDT CFF69802, 0574 (r1 AMD    POWERNOW  6040000 AMD         1)
[    0.000000] ACPI: SRAT CFF69D76, 0110 (r1 AMD    HAMMER    6040000 AMD         1)
[    0.000000] ACPI: SPCR CFF69E86, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL         1)
[    0.000000] ACPI: MCFG CFF69ED6, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
[    0.000000] ACPI: HPET CFF69F12, 0038 (r1 PTLTD  HPETTBL   6040000  LTP        1)
[    0.000000] ACPI: APIC CFF69F4A, 008E (r1 PTLTD       APIC    6040000  LTP        0)
[    0.000000] ACPI: BOOT CFF69FD8, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
[    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-d0000000
[    0.000000] SRAT: Node 0 PXM 0 100000000-130000000
[    0.000000] SRAT: Node 1 PXM 1 130000000-230000000
[    0.000000] NUMA: Allocated memnodemap from 11000 - 15680
[    0.000000] NUMA: Using 20 for the hash shift.
[    0.000000] Bootmem setup node 0 0000000000000000-0000000130000000
[    0.000000]   NODE_DATA [0000000000015680 - 000000000002067f]
[    0.000000]   bootmap [0000000000021000 -  0000000000046fff] pages 26
[    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0130000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
[    0.000000]   #2 [0000200000 - 0000f96910]    TEXT DATA BSS ==> [0000200000 - 0000f96910]
[    0.000000]   #3 [0037e0f000 - 0037fef850]          RAMDISK ==> [0037e0f000 - 0037fef850]
[    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
[    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE ==> [0000008000 - 000000c000]
[    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE ==> [000000c000 - 0000011000]
[    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP ==> [0000011000 - 0000015680]
[    0.000000] Bootmem setup node 1 0000000130000000-0000000230000000
[    0.000000]   NODE_DATA [0000000130000000 - 000000013000afff]
[    0.000000]   bootmap [000000013000b000 -  000000013002afff] pages 20
[    0.000000] (8 early reservations) ==> bootmem [0130000000 - 0230000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE
[    0.000000]   #2 [0000200000 - 0000f96910]    TEXT DATA BSS
[    0.000000]   #3 [0037e0f000 - 0037fef850]          RAMDISK
[    0.000000]   #4 [000009dc00 - 0000100000]    BIOS reserved
[    0.000000]   #5 [0000008000 - 000000c000]          PGTABLE
[    0.000000]   #6 [000000c000 - 0000011000]          PGTABLE
[    0.000000]   #7 [0000011000 - 0000015680]       MEMNODEMAP
[    0.000000] Scan SMP from ffff880000000000 for 1024 bytes.
[    0.000000] Scan SMP from ffff88000009fc00 for 1024 bytes.
[    0.000000] Scan SMP from ffff8800000f0000 for 65536 bytes.
[    0.000000] found SMP MP-table at [ffff8800000f7600] 000f7600
[    0.000000]  [ffffe20000000000-ffffe200071fffff] PMD -> [ffff880028200000-ffff88002e1fffff] on node 0
[    0.000000]  [ffffe20007200000-ffffe2000d1fffff] PMD -> [ffff880130200000-ffff8801361fffff] on node 1
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00230000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x0000009d
[    0.000000]     0: 0x00000100 -> 0x000cff60
[    0.000000]     0: 0x00100000 -> 0x00130000
[    0.000000]     1: 0x00130000 -> 0x00230000
[    0.000000] On node 0 totalpages: 1048317
[    0.000000]   DMA zone: 307 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 823232 pages, LIFO batch:31
[    0.000000]   Normal zone: 192000 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 1048576
[    0.000000]   Normal zone: 1024000 pages, LIFO batch:31
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x1008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 0, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Setting APIC routing to flat
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] mapped APIC to ffffffffff5fc000 (        fee00000)
[    0.000000] mapped IOAPIC to ffffffffff5fb000 (00000000fec00000)
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000ce000
[    0.000000] PM: Registered nosave memory: 00000000000ce000 - 0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000cff60000 - 00000000cff6a000
[    0.000000] PM: Registered nosave memory: 00000000cff6a000 - 00000000cff80000
[    0.000000] PM: Registered nosave memory: 00000000cff80000 - 00000000d0000000
[    0.000000] PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000
[    0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
[    0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
[    0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000
[    0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000
[    0.000000] PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
[    0.000000] PM: Registered nosave memory: 00000000fee01000 - 00000000fff80000
[    0.000000] PM: Registered nosave memory: 00000000fff80000 - 0000000100000000
[    0.000000] Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
[    0.000000] PERCPU: Allocating 51872 bytes of per cpu data
[    0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 2
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 2039539
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/ram0 real_root=/dev/sda3 init=/linuxrc debug nmi_watchdog=2 show_msr=1 apic=debug console=tty0 console=ttyS0,115200n8
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[    0.000000] Extended CMOS year: 2000
[    0.000000] TSC: PIT calibration confirmed by PMTIMER.
[    0.000000] TSC: using PIT calibration value
[    0.000000] Detected 2799.937 MHz processor.
[    0.003333] spurious 8259A interrupt: IRQ7.
[    0.003333] Console: colour VGA+ 80x25
[    0.003333] console [tty0] enabled
[    0.003333] console [ttyS0] enabled
[    0.003333] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.003333] ... MAX_LOCKDEP_SUBCLASSES:    8
[    0.003333] ... MAX_LOCK_DEPTH:          48
[    0.003333] ... MAX_LOCKDEP_KEYS:        8191
[    0.003333] ... CLASSHASH_SIZE:           4096
[    0.003333] ... MAX_LOCKDEP_ENTRIES:     8192
[    0.003333] ... MAX_LOCKDEP_CHAINS:      16384
[    0.003333] ... CHAINHASH_SIZE:          8192
[    0.003333]  memory used by lock dependency info: 3839 kB
[    0.003333]  per task-struct memory footprint: 1920 bytes
[    0.003333] Checking aperture...
[    0.003333] No AGP bridge found
[    0.003333] Node 0: aperture @ 0 size 32 MB
[    0.003333] Your BIOS doesn't leave a aperture memory hole
[    0.003333] Please enable the IOMMU option in the BIOS setup
[    0.003333] This costs you 64 MB of RAM
[    0.003333] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.003333] PM: Registered nosave memory: 0000000020000000 - 0000000024000000
[    0.003333] Memory: 8108304k/9175040k available (3897k kernel code, 279268k reserved, 2242k data, 676k init)
[    0.003333] CPA: page pool initialized 1 of 1 pages preallocated
[    0.003333] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.003333] hpet clockevent registered
[    0.003341] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.20 BogoMIPS (lpj=9333123)
[    0.010027] Security Framework initialized
[    0.016692] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.026447] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.029173] Mount-cache hash table entries: 256
[    0.030310] Initializing cgroup subsys ns
[    0.033338] Initializing cgroup subsys cpuacct
[    0.036668] Initializing cgroup subsys memory
[    0.040007] Initializing cgroup subsys devices
[    0.043340] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.046665] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.049999] CPU 0/0 -> Node 0
[    0.053332] tseg: 00cff80000
[    0.056665] CPU: Physical Processor ID: 0
[    0.059998] CPU: Processor Core ID: 0
[    0.063336] using C1E aware idle routine
[    0.066685] ACPI: Core revision 20080609
[    0.076723] Getting VERSION: 80050010
[    0.079997] Getting VERSION: 80050010
[    0.083330] Getting ID: 0
[    0.086663] Getting ID: ff000000
[    0.089994] Getting LVT0: 10000
[    0.089996] Getting LVT1: 10000
[    0.093330] masked ExtINT on CPU#0
[    0.096903] ENABLING IO-APIC IRQs
[    0.099995] init IO_APIC IRQs
[    0.103329]  IO-APIC (apicid-pin) 4-0 not connected.
[    0.108339] IOAPIC[0]: Set routing entry (4-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
[    0.110000] IOAPIC[0]: Set routing entry (4-2 -> 0x30 -> IRQ 0 Mode:0 Active:0)
[    0.113325] IOAPIC[0]: Set routing entry (4-3 -> 0x33 -> IRQ 3 Mode:0 Active:0)
[    0.113332] IOAPIC[0]: Set routing entry (4-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
[    0.116665] IOAPIC[0]: Set routing entry (4-5 -> 0x35 -> IRQ 5 Mode:0 Active:0)
[    0.119998] IOAPIC[0]: Set routing entry (4-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
[    0.123331] IOAPIC[0]: Set routing entry (4-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
[    0.126664] IOAPIC[0]: Set routing entry (4-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
[    0.129997] IOAPIC[0]: Set routing entry (4-9 -> 0x39 -> IRQ 9 Mode:1 Active:1)
[    0.133330] IOAPIC[0]: Set routing entry (4-10 -> 0x3a -> IRQ 10 Mode:0 Active:0)
[    0.136663] IOAPIC[0]: Set routing entry (4-11 -> 0x3b -> IRQ 11 Mode:0 Active:0)
[    0.139997] IOAPIC[0]: Set routing entry (4-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
[    0.143330] IOAPIC[0]: Set routing entry (4-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
[    0.146663] IOAPIC[0]: Set routing entry (4-14 -> 0x3e -> IRQ 14 Mode:0 Active:0)
[    0.149996] IOAPIC[0]: Set routing entry (4-15 -> 0x3f -> IRQ 15 Mode:0 Active:0)
[    0.153328]  IO-APIC (apicid-pin) 4-16, 4-17, 4-18, 4-19, 4-20, 4-21, 4-22, 4-23 not connected.
[    0.163485] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.200845] CPU0: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.206654] Using local APIC timer interrupts.
[    0.213320] APIC timer calibration result 12499727
[    0.216654] Detected 12.499 MHz APIC timer.
[    0.220144] lockdep: fixing up alternatives.
[    0.223342] Booting processor 1/1 ip 6000
[    0.003333] Initializing CPU#1
[    0.003333] masked ExtINT on CPU#1
[    0.003333] Calibrating delay using timer specific routine.. 5602.22 BogoMIPS (lpj=9333158)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 1/1 -> Node 0
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
[    0.320369] CPU1: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.330023] lockdep: fixing up alternatives.
[    0.333325] Booting processor 2/2 ip 6000
[    0.003333] Initializing CPU#2
[    0.003333] masked ExtINT on CPU#2
[    0.003333] Calibrating delay using timer specific routine.. 5602.25 BogoMIPS (lpj=9333200)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 2/2 -> Node 1
[    0.003333] CPU: Physical Processor ID: 1
[    0.003333] CPU: Processor Core ID: 0
[    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 0x7010600070106
[    0.430543] CPU2: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.440047] lockdep: fixing up alternatives.
[    0.443324] Booting processor 3/3 ip 6000
[    0.003333] Initializing CPU#3
[    0.003333] masked ExtINT on CPU#3
[    0.003333] Calibrating delay using timer specific routine.. 5602.23 BogoMIPS (lpj=9333179)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.003333] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.003333] CPU 3/3 -> Node 1
[    0.003333] CPU: Physical Processor ID: 1
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106
[    0.540532] CPU3: Dual-Core AMD Opteron(tm) Processor 2220 stepping 03
[    0.549981] Brought up 4 CPUs
[    0.553128] Total of 4 processors activated (22408.91 BogoMIPS).
[    0.553363] Testing NMI watchdog ... OK.
[    0.626894] net_namespace: 968 bytes
[    0.630205] xor: automatically using best checksumming function: generic_sse
[    0.649969]    generic_sse:  8630.400 MB/sec
[    0.653302] xor: using function: generic_sse (8630.400 MB/sec)
[    0.656696] NET: Registered protocol family 16
[    0.660253] No dock devices found.
[    0.663504] node 0 link 2: io port [1000, 3fff]
[    0.666636] TOM: 00000000d0000000 aka 3328M
[    0.669969] node 0 link 2: mmio [d0000000, dfffffff]
[    0.676635] node 0 link 2: mmio [a0000, bffff]
[    0.680343] node 0 link 2: mmio [e0000000, e7ffffff]
[    0.686635] TOM2: 0000000230000000 aka 8960M
[    0.689969] bus: [00,ff] on node 0 link 2
[    0.693307] bus: 00 index 0 io port: [0, ffff]
[    0.696635] bus: 00 index 1 mmio: [d0000000, ffffffff]
[    0.699968] bus: 00 index 2 mmio: [a0000, bffff]
[    0.703302] bus: 00 index 3 mmio: [230000000, fcffffffff]
[    0.706647] ACPI: bus type pci registered
[    0.710112] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 4
[    0.713303] PCI: MCFG area at e0000000 reserved in E820
[    0.716740] PCI: Using MMCONFIG at e0000000 - e04fffff
[    0.719968] PCI: Using configuration type 1 for base access
[    0.728437] ACPI: EC: Look up EC in DSDT
[    0.736968] ACPI: Interpreter enabled
[    0.739969] ACPI: (supports S0 S1 S3 S4 S5)
[    0.745030] ACPI: Using IOAPIC for interrupt routing
[    0.755036] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    0.756944] pci 0000:00:01.1: PME# supported from D3hot D3cold
[    0.759972] pci 0000:00:01.1: PME# disabled
[    0.763342] pci 0000:00:02.0: supports D1
[    0.766635] pci 0000:00:02.0: supports D2
[    0.769968] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.773303] pci 0000:00:02.0: PME# disabled
[    0.776673] pci 0000:00:02.1: supports D1
[    0.779968] pci 0000:00:02.1: supports D2
[    0.783302] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot D3cold
[    0.786636] pci 0000:00:02.1: PME# disabled
[    0.790205] pci 0000:00:08.0: supports D1
[    0.793301] pci 0000:00:08.0: supports D2
[    0.796635] pci 0000:00:08.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.799971] pci 0000:00:08.0: PME# disabled
[    0.803350] pci 0000:00:09.0: supports D1
[    0.806635] pci 0000:00:09.0: supports D2
[    0.809968] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.813304] pci 0000:00:09.0: PME# disabled
[    0.816667] pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.819969] pci 0000:00:0a.0: PME# disabled
[    0.823336] pci 0000:00:0d.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.826636] pci 0000:00:0d.0: PME# disabled
[    0.829998] pci 0000:00:0f.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.833302] pci 0000:00:0f.0: PME# disabled
[    0.836887] pci 0000:01:04.0: supports D1
[    0.839968] pci 0000:01:04.0: supports D2
[    0.846644] pci 0000:00:06.0: transparent bridge
[    0.849969] PCI: bridge 0000:00:06.0 io port: [3000, 3fff]
[    0.853303] PCI: bridge 0000:00:06.0 32bit mmio: [d0100000, d01fffff]
[    0.856636] PCI: bridge 0000:00:06.0 32bit mmio pref: [d8000000, dfffffff]
[    0.860036] pci 0000:02:00.0: supports D1
[    0.863330] PCI: bridge 0000:00:0a.0 32bit mmio: [d0200000, d02fffff]
[    0.866721] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.870040] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT]
[    0.873324] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT]
[    0.876660] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR2._PRT]
[    0.879993] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR5._PRT]
[    0.908235] ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.918928] ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.925216] ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.932886] ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.940495] ACPI: PCI Interrupt Link [LK1E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.951499] ACPI: PCI Interrupt Link [LK2E] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    0.960946] ACPI: PCI Interrupt Link [LK3E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.971949] ACPI: PCI Interrupt Link [LK4E] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.979551] ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    0.987162] ACPI: PCI Interrupt Link [LUS0] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    0.996787] ACPI: PCI Interrupt Link [LMA2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    1.006297] ACPI: PCI Interrupt Link [LUS2] (IRQs *5 7 10 11 14 15 16 17 18 19 20 21 22 23)
[    1.012659] ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    1.019013] ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    1.026787] ACPI: PCI Interrupt Link [LPID] (IRQs 5 7 10 11 14 15 16 17 18 19 20 21 22 23) *0, disabled.
[    1.037162] ACPI: PCI Interrupt Link [LTID] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    1.047300] ACPI: PCI Interrupt Link [LSI1] (IRQs 5 7 *10 11 14 15 16 17 18 19 20 21 22 23)
[    1.056787] ACPI: PCI Interrupt Link [LSI2] (IRQs 5 7 10 *11 14 15 16 17 18 19 20 21 22 23)
[    1.066914] Linux Plug and Play Support v0.97 (c) Adam Belay
[    1.070030] pnp: PnP ACPI init
[    1.073313] ACPI: bus type pnp registered
[    1.077375] IOAPIC[0]: Set routing entry (4-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
[    1.080619] IOAPIC[0]: Set routing entry (4-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
[    1.083402] IOAPIC[0]: Set routing entry (4-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
[    1.087135] IOAPIC[0]: Set routing entry (4-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
[    1.093699] IOAPIC[0]: Set routing entry (4-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
[    1.097564] IOAPIC[0]: Set routing entry (4-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
[    1.104093] pnp: PnP ACPI: found 14 devices
[    1.106636] ACPI: ACPI bus type pnp unregistered
[    1.113565] SCSI subsystem initialized
[    1.116800] libata version 3.00 loaded.
[    1.120253] usbcore: registered new interface driver usbfs
[    1.123369] usbcore: registered new interface driver hub
[    1.130010] usbcore: registered new device driver usb
[    1.133672] PCI: Using ACPI for IRQ routing
[    1.136637] number of MP IRQ sources: 16.
[    1.139968] number of IO-APIC #4 registers: 24.
[    1.143301] testing the IO APIC.......................
[    1.146640] 
[    1.148314] IO APIC #4......
[    1.149968] .... register #00: 04000000
[    1.153302] .......    : physical APIC id: 04
[    1.156635] .... register #01: 00170011
[    1.159968] .......     : max redirection entries: 0017
[    1.163301] .......     : PRQ implemented: 0
[    1.166635] .......     : IO APIC version: 0011
[    1.169968] .... register #02: 04000000
[    1.173301] .......     : arbitration: 04
[    1.176635] .... IRQ redirection table:
[    1.179968]  NR Dst Mask Trig IRR Pol Stat Dmod Deli Vect:   
[    1.183305]  00 000 1    0    0   0   0    0    0    00
[    1.186638]  01 00F 0    0    0   0   0    1    1    31
[    1.193305]  02 00F 0    0    0   0   0    1    1    30
[    1.199971]  03 00F 0    0    0   0   0    1    1    33
[    1.203305]  04 00F 0    0    0   0   0    1    1    34
[    1.209971]  05 00F 1    0    0   0   0    1    1    35
[    1.216638]  06 00F 0    0    0   0   0    1    1    36
[    1.219971]  07 00F 0    0    0   0   0    1    1    37
[    1.226638]  08 00F 0    0    0   0   0    1    1    38
[    1.233305]  09 00F 0    1    0   1   0    1    1    39
[    1.236638]  0a 00F 1    0    0   0   0    1    1    3A
[    1.243305]  0b 00F 1    0    0   0   0    1    1    3B
[    1.249971]  0c 00F 0    0    0   0   0    1    1    3C
[    1.253305]  0d 00F 0    0    0   0   0    1    1    3D
[    1.259971]  0e 00F 0    0    0   0   0    1    1    3E
[    1.263305]  0f 00F 0    0    0   0   0    1    1    3F
[    1.269974]  10 000 1    0    0   0   0    0    0    00
[    1.276638]  11 000 1    0    0   0   0    0    0    00
[    1.279971]  12 000 1    0    0   0   0    0    0    00
[    1.286638]  13 000 1    0    0   0   0    0    0    00
[    1.293305]  14 000 1    0    0   0   0    0    0    00
[    1.296638]  15 000 1    0    0   0   0    0    0    00
[    1.303305]  16 000 1    0    0   0   0    0    0    00
[    1.309971]  17 000 1    0    0   0   0    0    0    00
[    1.313301] IRQ to pin mappings:
[    1.316635] IRQ0 -> 0:2
[    1.320918] IRQ1 -> 0:1
[    1.324251] IRQ3 -> 0:3
[    1.327009] IRQ4 -> 0:4
[    1.329968] IRQ5 -> 0:5
[    1.332701] IRQ6 -> 0:6
[    1.334254] IRQ7 -> 0:7
[    1.337587] IRQ8 -> 0:8
[    1.340914] IRQ9 -> 0:9
[    1.343676] IRQ10 -> 0:10
[    1.346635] IRQ11 -> 0:11
[    1.349540] IRQ12 -> 0:12
[    1.350999] IRQ13 -> 0:13
[    1.354339] IRQ14 -> 0:14
[    1.357670] IRQ15 -> 0:15
[    1.361004] .................................... done.
[    1.376899] PCI-DMA: Disabling AGP.
[    1.381108] PCI-DMA: aperture base @ 20000000 size 65536 KB
[    1.383302] PCI-DMA: using GART IOMMU.
[    1.386637] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    1.390525] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
[    1.397011] hpet0: 3 32-bit timers, 25000000 Hz
[    1.401065] ACPI: RTC can wake from S4
[    1.403313] Switched to high resolution mode on CPU 0
[    1.403376] Switched to high resolution mode on CPU 2
[    1.403396] Switched to high resolution mode on CPU 3
[    1.406608] Switched to high resolution mode on CPU 1
[    1.436703] system 00:00: iomem range 0xffc00000-0xffffffff could not be reserved
[    1.444539] system 00:00: iomem range 0xfec00000-0xfec00fff could not be reserved
[    1.452372] system 00:00: iomem range 0xfee00000-0xfeefffff could not be reserved
[    1.460206] system 00:00: iomem range 0xfed00000-0xfed00fff has been reserved
[    1.467553] system 00:03: iomem range 0xd0000000-0xd0007fff has been reserved
[    1.474888] system 00:04: ioport range 0x1000-0x107f has been reserved
[    1.481616] system 00:04: ioport range 0x1080-0x10ff has been reserved
[    1.488341] system 00:04: ioport range 0x1400-0x147f has been reserved
[    1.495066] system 00:04: ioport range 0x1480-0x14ff has been reserved
[    1.501791] system 00:04: ioport range 0x1800-0x187f has been reserved
[    1.508520] system 00:04: ioport range 0x1880-0x18ff has been reserved
[    1.515244] system 00:04: ioport range 0x2440-0x247f has been reserved
[    1.521967] system 00:04: ioport range 0x2400-0x243f has been reserved
[    1.528705] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
[    1.535252] system 00:07: ioport range 0xc05-0xc06 has been reserved
[    1.542958] pci 0000:00:06.0: PCI bridge, secondary bus 0000:01
[    1.549075] pci 0000:00:06.0:   IO window: 0x3000-0x3fff
[    1.554587] pci 0000:00:06.0:   MEM window: 0xd0100000-0xd01fffff
[    1.560884] pci 0000:00:06.0:   PREFETCH window: 0x000000d8000000-0x000000dfffffff
[    1.568802] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:02
[    1.574919] pci 0000:00:0a.0:   IO window: disabled
[    1.579997] pci 0000:00:0a.0:   MEM window: 0xd0200000-0xd02fffff
[    1.586290] pci 0000:00:0a.0:   PREFETCH window: 0x000000d1000000-0x000000d10fffff
[    1.594222] pci 0000:00:0d.0: PCI bridge, secondary bus 0000:03
[    1.600336] pci 0000:00:0d.0:   IO window: disabled
[    1.605406] pci 0000:00:0d.0:   MEM window: disabled
[    1.610562] pci 0000:00:0d.0:   PREFETCH window: disabled
[    1.616154] pci 0000:00:0f.0: PCI bridge, secondary bus 0000:04
[    1.622273] pci 0000:00:0f.0:   IO window: disabled
[    1.627350] pci 0000:00:0f.0:   MEM window: disabled
[    1.632508] pci 0000:00:0f.0:   PREFETCH window: disabled
[    1.638115] pci 0000:00:06.0: setting latency timer to 64
[    1.643717] pci 0000:00:0a.0: setting latency timer to 64
[    1.654201] pci 0000:00:0d.0: setting latency timer to 64
[    1.659803] pci 0000:00:0f.0: setting latency timer to 64
[    1.665399] bus: 00 index 0 io port: [0, ffff]
[    1.670040] bus: 00 index 1 mmio: [0, ffffffffffffffff]
[    1.675458] bus: 01 index 0 io port: [3000, 3fff]
[    1.680356] bus: 01 index 1 mmio: [d0100000, d01fffff]
[    1.685687] bus: 01 index 2 mmio: [d8000000, dfffffff]
[    1.691027] bus: 01 index 3 io port: [0, ffff]
[    1.695669] bus: 01 index 4 mmio: [0, ffffffffffffffff]
[    1.701085] bus: 02 index 0 mmio: [0, 0]
[    1.705201] bus: 02 index 1 mmio: [d0200000, d02fffff]
[    1.710532] bus: 02 index 2 mmio: [d1000000, d10fffff]
[    1.715862] bus: 02 index 3 mmio: [0, 0]
[    1.719976] bus: 03 index 0 mmio: [0, 0]
[    1.724093] bus: 03 index 1 mmio: [0, 0]
[    1.728211] bus: 03 index 2 mmio: [0, 0]
[    1.732327] bus: 03 index 3 mmio: [0, 0]
[    1.736443] bus: 04 index 0 mmio: [0, 0]
[    1.740562] bus: 04 index 1 mmio: [0, 0]
[    1.744683] bus: 04 index 2 mmio: [0, 0]
[    1.748801] bus: 04 index 3 mmio: [0, 0]
[    1.752934] NET: Registered protocol family 2
[    1.790275] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    1.800242] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    1.813278] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
[    1.822932] TCP: Hash tables configured (established 524288 bind 65536)
[    1.829770] TCP reno registered
[    1.840164] NET: Registered protocol family 1
[    1.844853] checking if image is initramfs... it is
[    1.946863] Freeing initrd memory: 1922k freed
[    1.952558] Simple Boot Flag at 0x37 set to 0x80
[    1.960282] audit: initializing netlink socket (disabled)
[    1.965948] type=2000 audit(1221684197.964:1): initialized
[    1.977176] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.987996] VFS: Disk quotas dquot_6.5.1
[    1.992322] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.000631] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    2.007574] msgmni has been set to 15840
[    2.011926] async_tx: api initialized (async)
[    2.016496] io scheduler noop registered
[    2.020626] io scheduler anticipatory registered
[    2.025443] io scheduler deadline registered
[    2.030096] io scheduler cfq registered (default)
[    2.035040] pci 0000:00:00.0: Enabling HT MSI Mapping
[    2.242753] pci 0000:00:05.0: Enabling HT MSI Mapping
[    2.248026] pci 0000:00:05.1: Enabling HT MSI Mapping
[    2.253298] pci 0000:00:05.2: Enabling HT MSI Mapping
[    2.258569] pci 0000:00:06.0: Enabling HT MSI Mapping
[    2.263840] pci 0000:00:08.0: Enabling HT MSI Mapping
[    2.269115] pci 0000:00:09.0: Enabling HT MSI Mapping
[    2.274384] pci 0000:00:0a.0: Enabling HT MSI Mapping
[    2.279654] pci 0000:00:0d.0: Enabling HT MSI Mapping
[    2.284923] pci 0000:00:0f.0: Enabling HT MSI Mapping
[    2.290198] pci 0000:01:04.0: Boot video device
[    2.295130] pcieport-driver 0000:00:0a.0: setting latency timer to 64
[    2.301796] pcieport-driver 0000:00:0a.0: found MSI capability
[    2.307876] pci_express 0000:00:0a.0:pcie00: allocate port service
[    2.314364] pci_express 0000:00:0a.0:pcie01: allocate port service
[    2.320814] pci_express 0000:00:0a.0:pcie03: allocate port service
[    2.327310] pcieport-driver 0000:00:0d.0: setting latency timer to 64
[    2.333972] pcieport-driver 0000:00:0d.0: found MSI capability
[    2.340033] pci_express 0000:00:0d.0:pcie00: allocate port service
[    2.346478] pci_express 0000:00:0d.0:pcie01: allocate port service
[    2.352928] pci_express 0000:00:0d.0:pcie03: allocate port service
[    2.359412] pcieport-driver 0000:00:0f.0: setting latency timer to 64
[    2.366074] pcieport-driver 0000:00:0f.0: found MSI capability
[    2.372130] pci_express 0000:00:0f.0:pcie00: allocate port service
[    2.378570] pci_express 0000:00:0f.0:pcie01: allocate port service
[    2.385024] pci_express 0000:00:0f.0:pcie03: allocate port service
[    2.393252] aer 0000:00:0a.0:pcie01: AER service couldn't init device: no _OSC support
[    2.403151] aer 0000:00:0d.0:pcie01: AER service couldn't init device: no _OSC support
[    2.413036] aer 0000:00:0f.0:pcie01: AER service couldn't init device: no _OSC support
[    2.421764] input: Power Button (FF) as /class/input/input0
[    2.427547] ACPI: Power Button (FF) [PWRF]
[    2.431956] input: Power Button (CM) as /class/input/input1
[    2.437730] ACPI: Power Button (CM) [PWRB]
[    2.442465] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[    2.448539] processor ACPI0007:00: registered as cooling_device0
[    2.454750] ACPI: Processor [C000] (supports 2 throttling states)
[    2.461243] processor ACPI0007:01: registered as cooling_device1
[    2.467553] processor ACPI0007:02: registered as cooling_device2
[    2.473856] processor ACPI0007:03: registered as cooling_device3
[    2.551196] hpet_resources: 0xfed00000 is busy
[    2.556037] Non-volatile memory driver v1.2
[    2.560567] Linux agpgart interface v0.103
[    2.564963] Serial: 8250/16550 driver4 ports, IRQ sharing disabled
[    2.571547] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.578525] 00:0c: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.584655] Floppy drive(s): fd0 is 1.44M
[    5.600023] floppy0: no floppy controllers found
[    5.607494] brd: module loaded
[    5.611934] loop: module loaded
[    5.615273] tun: Universal TUN/TAP device driver, 1.6
[    5.620523] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    5.627097] console [netcon0] enabled
[    5.630964] netconsole: network logging started
[    5.635691] Uniform Multi-Platform E-IDE driver
[    5.640554] ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
[    5.650661] Probing IDE interface ide0...
[    6.623534] hdb: CD-224E-N, ATAPI CD/DVD-ROM drive
[    6.680121] Probing IDE interface ide1...
[    7.220119] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[    7.224942] ide1 at 0x170-0x177,0x376 on irq 15
[    7.231765] hdb: ATAPI 24X CD-ROM drive, 256kB Cache
[    7.237145] Uniform CD-ROM driver Revision: 3.20
[    7.252071] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
[    7.259881] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[    7.267154] megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008
[    7.273348] Driver 'sd' needs updating - please use bus_type methods
[    7.279972] Driver 'sr' needs updating - please use bus_type methods
[    7.287151] sata_nv 0000:00:05.0: version 3.5
[    7.291999] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23
[    7.297948] IOAPIC[0]: Set routing entry (4-23 -> 0x61 -> IRQ 23 Mode:1 Active:1)
[    7.305794] sata_nv 0000:00:05.0: PCI INT A -> Link[LTID] -> GSI 23 (level, low) -> IRQ 23
[    7.314408] sata_nv 0000:00:05.0: Using SWNCQ mode
[    7.319523] sata_nv 0000:00:05.0: setting latency timer to 64
[    7.325654] scsi0 : sata_nv
[    7.328840] scsi1 : sata_nv
[    7.332023] ata1: SATA max UDMA/133 cmd 0x24d0 ctl 0x24c4 bmdma 0x2490 irq 23
[    7.339356] ata2: SATA max UDMA/133 cmd 0x24c8 ctl 0x24c0 bmdma 0x2498 irq 23
[    7.677046] ata1: SATA link down (SStatus 0 SControl 300)
[    8.010394] ata2: SATA link down (SStatus 0 SControl 300)
[    8.016279] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22
[    8.022230] IOAPIC[0]: Set routing entry (4-22 -> 0x69 -> IRQ 22 Mode:1 Active:1)
[    8.030072] sata_nv 0000:00:05.1: PCI INT B -> Link[LSI1] -> GSI 22 (level, low) -> IRQ 22
[    8.038685] sata_nv 0000:00:05.1: Using SWNCQ mode
[    8.043786] sata_nv 0000:00:05.1: setting latency timer to 64
[    8.049864] scsi2 : sata_nv
[    8.052998] scsi3 : sata_nv
[    8.056180] ata3: SATA max UDMA/133 cmd 0x24e8 ctl 0x24dc bmdma 0x24a0 irq 22
[    8.063516] ata4: SATA max UDMA/133 cmd 0x24e0 ctl 0x24d8 bmdma 0x24a8 irq 22
[    8.400336] ata3: SATA link down (SStatus 0 SControl 300)
[    8.733666] ata4: SATA link down (SStatus 0 SControl 300)
[    8.739534] ACPI: PCI Interrupt Link [LSI2] enabled at IRQ 21
[    8.745490] IOAPIC[0]: Set routing entry (4-21 -> 0x71 -> IRQ 21 Mode:1 Active:1)
[    8.753340] sata_nv 0000:00:05.2: PCI INT C -> Link[LSI2] -> GSI 21 (level, low) -> IRQ 21
[    8.761952] sata_nv 0000:00:05.2: Using SWNCQ mode
[    8.767051] sata_nv 0000:00:05.2: setting latency timer to 64
[    8.773139] scsi4 : sata_nv
[    8.776273] scsi5 : sata_nv
[    8.779449] ata5: SATA max UDMA/133 cmd 0x2800 ctl 0x24f4 bmdma 0x24b0 irq 21
[    8.786784] ata6: SATA max UDMA/133 cmd 0x24f8 ctl 0x24f0 bmdma 0x24b8 irq 21
[    9.123711] ata5: SATA link down (SStatus 0 SControl 300)
[    9.457043] ata6: SATA link down (SStatus 0 SControl 300)
[    9.462786] pata_amd 0000:00:04.0: version 0.3.10
[    9.467719] pata_amd 0000:00:04.0: BAR 0: can't reserve I/O region [0x1f0-0x1f7]
[    9.475461] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 0 (errno=-16)
[    9.483729] pata_amd 0000:00:04.0: BAR 2: can't reserve I/O region [0x170-0x177]
[    9.491476] pata_amd 0000:00:04.0: failed to request/iomap BARs for port 1 (errno=-16)
[    9.499744] pata_amd 0000:00:04.0: no available native port
[    9.505848] Fusion MPT base driver 3.04.07
[    9.510141] Copyright (c) 1999-2008 LSI Corporation
[    9.515225] Fusion MPT SPI Host driver 3.04.07
[    9.519953] Fusion MPT SAS Host driver 3.04.07
[    9.525373] ieee1394: raw1394: /dev/raw1394 device initialized
[    9.531907] ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 20
[    9.537855] IOAPIC[0]: Set routing entry (4-20 -> 0x79 -> IRQ 20 Mode:1 Active:1)
[    9.545699] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUS2] -> GSI 20 (level, low) -> IRQ 20
[    9.554409] ehci_hcd 0000:00:02.1: setting latency timer to 64
[    9.560437] ehci_hcd 0000:00:02.1: EHCI Host Controller
[    9.566111] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
[    9.578754] ehci_hcd 0000:00:02.1: debug port 1
[    9.583480] ehci_hcd 0000:00:02.1: cache line size of 64 is not supported
[    9.590474] ehci_hcd 0000:00:02.1: irq 20, io mem 0xd0041000
[    9.603347] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[    9.611332] usb usb1: configuration #1 chosen from 1 choice
[    9.617202] hub 1-0:1.0: USB hub found
[    9.621170] hub 1-0:1.0: 10 ports detected
[    9.727087] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
[    9.735198] ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 19
[    9.741141] IOAPIC[0]: Set routing entry (4-19 -> 0x81 -> IRQ 19 Mode:1 Active:1)
[    9.748986] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUS0] -> GSI 19 (level, low) -> IRQ 19
[    9.757691] ohci_hcd 0000:00:02.0: setting latency timer to 64
[    9.763725] ohci_hcd 0000:00:02.0: OHCI Host Controller
[    9.769310] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
[    9.777074] ohci_hcd 0000:00:02.0: irq 19, io mem 0xd0040000
[    9.835493] usb usb2: configuration #1 chosen from 1 choice
[    9.841355] hub 2-0:1.0: USB hub found
[    9.845307] hub 2-0:1.0: 10 ports detected
[    9.950334] USB Universal Host Controller Interface driver v3.0
[    9.956623] usbcore: registered new interface driver usblp
[    9.962303] Initializing USB Mass Storage driver...
[    9.967449] usbcore: registered new interface driver usb-storage
[    9.973658] USB Mass Storage support registered.
[    9.978631] PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSE0] at 0x60,0x64 irq 1,12
[   10.236985] serio: i8042 KBD port at 0x60,0x64 irq 1
[   10.242370] mice: PS/2 mouse device common for all mice
[   10.353338] md: linear personality registered for level -1
[   10.447540] md: raid0 personality registered for level 0
[   10.453050] md: raid1 personality registered for level 1
[   10.458553] md: raid10 personality registered for level 10
[   10.520442] raid6: int64x1   2568 MB/s
[   10.580428] raid6: int64x2   3277 MB/s
[   10.640437] raid6: int64x4   3227 MB/s
[   10.700434] raid6: int64x8   2278 MB/s
[   10.760424] raid6: sse2x1    3921 MB/s
[   10.820421] raid6: sse2x2    5110 MB/s
[   10.880420] raid6: sse2x4    5274 MB/s
[   10.884370] raid6: using algorithm sse2x4 (5274 MB/s)
[   10.889617] md: raid6 personality registered for level 6
[   10.895129] md: raid5 personality registered for level 5
[   10.900642] md: raid4 personality registered for level 4
[   10.906153] md: multipath personality registered for level -4
[   10.912349] device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
[   10.921284] cpuidle: using governor ladder
[   10.925801] cpuidle: using governor menu
[   10.930007] usbcore: registered new interface driver usbhid
[   10.935775] usbhid: v2.6:USB HID core driver
[   10.940354] oprofile: using NMI interrupt.
[   10.945133] TCP cubic registered
[   10.948765] NET: Registered protocol family 10
[   10.954418] IPv6 over IPv4 tunneling driver
[   10.959568] NET: Registered protocol family 17
[   10.964611] RPC: Registered udp transport module.
[   10.969516] RPC: Registered tcp transport module.
[   10.974504] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2220 processors (4 cpu cores) (version 2.20.00)
[   10.980007] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
[   10.991352] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
[   10.997210] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
[   11.003068] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
[   11.008840] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
[   11.014696] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
[   11.020559] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
[   10.980842] powernow-k8:    0 : fid 0x14 (2800 MHz), vid 0x8
[   11.032498] powernow-k8:    1 : fid 0x12 (2600 MHz), vid 0xa
[   11.038362] powernow-k8:    2 : fid 0x10 (2400 MHz), vid 0xc
[   11.044220] powernow-k8:    3 : fid 0xe (2200 MHz), vid 0xe
[   11.049992] powernow-k8:    4 : fid 0xc (2000 MHz), vid 0x10
[   11.055850] powernow-k8:    5 : fid 0xa (1800 MHz), vid 0x10
[   11.061710] powernow-k8:    6 : fid 0x2 (1000 MHz), vid 0x12
[   11.068400] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[   11.074956] Freeing unused kernel memory: 676k freed
[   11.080499] Write protecting the kernel read-only data: 5672k
[   11.500848] Clocksource tsc unstable (delta = -74982452 ns)
[   11.622045] ACPI: PCI Interrupt Link [LK3E] enabled at IRQ 18
[   11.622054] IOAPIC[0]: Set routing entry (4-18 -> 0x89 -> IRQ 18 Mode:1 Active:1)
[   11.622117] arcmsr 0000:02:00.0: PCI INT A -> Link[LK3E] -> GSI 18 (level, low) -> IRQ 18
[   11.622124] arcmsr 0000:02:00.0: setting latency timer to 64
[   11.633781] ARECA RAID ADAPTER6: FIRMWARE VERSION V1.44 2008-1-31  
[   11.647101] scsi6 : Areca SATA Host Adapter RAID Controller( RAID6 capable)
[   11.647103]  Driver Version 1.20.00.15 2008/02/27
[   11.648067] scsi 6:0:0:0: Direct-Access     Areca    ARC-1280-VOL#00  R001 PQ: 0 ANSI: 5
[   11.649251] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[   11.649299] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
[   11.649347] sd 6:0:0:0: [sda] Write Protect is off
[   11.649350] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
[   11.649433] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.649633] sd 6:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[   11.649678] sd 6:0:0:0: [sda] 21484367872 512-byte hardware sectors (10999996 MB)
[   11.649720] sd 6:0:0:0: [sda] Write Protect is off
[   11.649723] sd 6:0:0:0: [sda] Mode Sense: cb 00 00 08
[   11.649805] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.649809]  sda: sda1 sda2 sda3 sda4
[   11.681499] sd 6:0:0:0: [sda] Attached SCSI disk
[   11.682282] sd 6:0:0:0: Attached scsi generic sg0 type 0
[   11.683553] scsi 6:0:16:0: Processor         Areca    RAID controller  R001 PQ: 0 ANSI: 0
[   11.684441] scsi 6:0:16:0: Attached scsi generic sg1 type 3
[   11.859757] 3ware Storage Controller device driver for Linux v1.26.02.002.
[   11.891003] 3ware 9000 Storage Controller device driver for Linux v2.26.02.011.
[   11.956151] Adaptec aacraid driver 1.1-5[2456]-ms
[   12.396745] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
[   12.404532] SGI XFS Quota Management subsystem
[   12.457945] Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
[   12.457954] Copyright (c) 1999-2006 Intel Corporation.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ???
  2008-09-17 20:48       ` j_kernel
@ 2008-09-18  7:09         ` Cyrill Gorcunov
  0 siblings, 0 replies; 20+ messages in thread
From: Cyrill Gorcunov @ 2008-09-18  7:09 UTC (permalink / raw)
  To: j_kernel
  Cc: Ingo Molnar, Andrew Morton, linux-kernel, bugme-daemon,
	Thomas Gleixner

[j_kernel@hoblitt.com - Wed, Sep 17, 2008 at 10:48:50AM -1000]
| On Wed, Sep 17, 2008 at 08:27:04PM +0400, Cyrill Gorcunov wrote:
| > Joshua, could you please do one more thing - boot the kernel
| > with "debug apic=debug nmi_watchdog=1" (among others) and
| > publish dmsg?
| 
| dmesg from "debug apic=debug nmi_watchdog=2" is attached.
|                                           ^
| 
| -J
| 
| --
...

No Joshua, exactly 'nmi_watchdog=1' since it's important to
find who is responsible that cpu is stuck while testing nmi
watchdog. 'debug and apic=debug' will show additional info
what is happening. And it would be just great if you able
to try latest -tip/master

	http://people.redhat.com/mingo/tip.git/README

since there a lot of work/fix are done. If it's a problem
to fetch this kernel thru git - me or Ingo (I suppose)
could make tar.bz2 archive so you'll not need to fetch
the whole history. I wouldn't say I'm really specialist
in this area so your report could be quite important
not for me only but for others too.

		- Cyrill -

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2008-09-18  7:10 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-09-17  2:43 [Bugme-new] [Bug 11543] New: kernel panic: softlockup in tick_periodic() ??? j_kernel
2008-09-17  7:13 ` Cyrill Gorcunov
2008-09-17  9:20   ` j_kernel
2008-09-17  9:38     ` Cyrill Gorcunov
2008-09-17 12:59     ` Cyrill Gorcunov
2008-09-17 13:13       ` Ingo Molnar
2008-09-17 13:29         ` Cyrill Gorcunov
2008-09-17 16:27     ` Cyrill Gorcunov
2008-09-17 20:48       ` j_kernel
2008-09-18  7:09         ` Cyrill Gorcunov
     [not found] <bug-11543-10286@http.bugzilla.kernel.org/>
2008-09-12  0:02 ` Andrew Morton
2008-09-12  2:54   ` j_kernel
2008-09-12  2:57     ` Andrew Morton
2008-09-12  9:13   ` Ingo Molnar
2008-09-13  0:13     ` Joshua Hoblitt
2008-09-15 21:06       ` Joshua Hoblitt
2008-09-16  2:54         ` Joshua Hoblitt
2008-09-16 14:14           ` Thomas Gleixner
2008-09-16 17:56             ` Cyrill Gorcunov
2008-09-16 17:57               ` Cyrill Gorcunov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox