[bug] 2.6.19-rc3 oops at __drain_pages during cpu hotplug tests + lockdep warning with xfs

[Ravikiran G Thirumalai <kiran@scalex86.org>]

[-- Attachment #1: Type: text/plain, Size: 5098 bytes --]

2.6.19-rc2mm2 as well as 2.6.19-rc3 panics with the following OOPS when I
run kernbench continuously and execute cpu offline and online in parallel. 
This is a 2 socket 4 core tyan opteron system with 8Gs of RAM.  
The kernel has lockdep, and other debug flags enabled.  Also, we have a 
lockdep warning with XFS.  Attaching the dmesg and the .config used as well.  
Here's the snapshot of lockdep warning + oops.


[  469.336719] 
[  469.336721] =============================================
[  469.336728] [ INFO: possible recursive locking detected ]
[  469.336731] 2.6.19-rc3 #2
[  469.336737] ---------------------------------------------
[  469.336745] rm/4538 is trying to acquire lock:
[  469.336753]  (&(&ip->i_lock)->mr_lock){----}, at: [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336782] 
[  469.336784] but task is already holding lock:
[  469.336791]  (&(&ip->i_lock)->mr_lock){----}, at: [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336816] 
[  469.336817] other info that might help us debug this:
[  469.336825] 3 locks held by rm/4538:
[  469.336832]  #0:  (&inode->i_mutex/1){--..}, at: [<ffffffff80286055>] do_unlinkat+0x75/0x136
[  469.336866]  #1:  (&inode->i_mutex){--..}, at: [<ffffffff80285f98>] vfs_unlink+0x45/0x8d
[  469.336897]  #2:  (&(&ip->i_lock)->mr_lock){----}, at: [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336928] 
[  469.336929] stack backtrace:
[  469.336936] 
[  469.336938] Call Trace:
[  469.336947]  [<ffffffff8024b4cc>] __lock_acquire+0x381/0xb8c
[  469.336957]  [<ffffffff8024bf80>] lock_acquire+0x4b/0x66
[  469.336966]  [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336976]  [<ffffffff80247c97>] down_write+0x1e/0x27
[  469.336985]  [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336993]  [<ffffffff8033cad1>] xfs_lock_dir_and_entry+0x92/0xce
[  469.337002]  [<ffffffff8033cd23>] xfs_remove+0x216/0x44a
[  469.337013]  [<ffffffff80346c96>] xfs_vn_unlink+0x21/0x4f
[  469.337024]  [<ffffffff804a72b9>] __mutex_lock_slowpath+0x249/0x279
[  469.337034]  [<ffffffff8024aaaf>] mark_held_locks+0x65/0x84
[  469.337043]  [<ffffffff804a72b9>] __mutex_lock_slowpath+0x249/0x279
[  469.337053]  [<ffffffff8024ac7c>] trace_hardirqs_on+0x101/0x128
[  469.337062]  [<ffffffff804a72dc>] __mutex_lock_slowpath+0x26c/0x279
[  469.337072]  [<ffffffff802832d9>] permission+0xa0/0xa9
[  469.337081]  [<ffffffff80285fb4>] vfs_unlink+0x61/0x8d
[  469.337090]  [<ffffffff80286098>] do_unlinkat+0xb8/0x136
[  469.337100]  [<ffffffff804a8649>] trace_hardirqs_on_thunk+0x35/0x37
[  469.337109]  [<ffffffff8024ac7c>] trace_hardirqs_on+0x101/0x128
[  469.337118]  [<ffffffff804a8649>] trace_hardirqs_on_thunk+0x35/0x37
[  469.337129]  [<ffffffff80209a4e>] system_call+0x7e/0x83
[  469.337137] 


[  488.264267] Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP: 
[  488.264271]  [<ffffffff8025d255>] __drain_pages+0x33/0x7d
[  488.264288] PGD 2753ae067 PUD 275f66067 PMD 0 
[  488.264302] Oops: 0000 [1] PREEMPT SMP 
[  488.264315] CPU 2 
[  488.264323] Modules linked in:
[  488.264332] Pid: 7617, comm: sh Not tainted 2.6.19-rc3 #2
[  488.264339] RIP: 0010:[<ffffffff8025d255>]  [<ffffffff8025d255>] __drain_pages+0x33/0x7d
[  488.264353] RSP: 0018:ffff81016dd3fdf8  EFLAGS: 00010046
[  488.264358] RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000000
[  488.264366] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffff810180000000
[  488.264373] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000001
[  488.264381] R10: ffffffff8025cb12 R11: 0000000000000000 R12: ffff810180000000
[  488.264389] R13: 0000000000000001 R14: 0000000000000001 R15: ffff81027a7e5998
[  488.264397] FS:  00002af4d80a5ae0(0000) GS:ffff81018001ad18(0000) knlGS:0000000000000000
[  488.264405] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  488.264412] CR2: 0000000000000000 CR3: 00000002767bf000 CR4: 00000000000006a0
[  488.264420] Process sh (pid: 7617, threadinfo ffff81016dd3e000, task ffff81016e03d080)
[  488.264427] Stack:  0000000000000001 0000000000000001 0000000000000007 0000000000000001
[  488.264448]  000000000000000f ffffffff8025e771 ffffffff80538860 ffffffff8023e3c2
[  488.264466]  0000000000000001 0000000000000001 ffff81016e11a0c0 ffffffff8024ff50
[  488.264481] Call Trace:
[  488.264490]  [<ffffffff8025e771>] page_alloc_cpu_notify+0x18/0x33
[  488.264500]  [<ffffffff8023e3c2>] notifier_call_chain+0x23/0x32
[  488.264509]  [<ffffffff8024ff50>] _cpu_down+0x184/0x245
[  488.264516]  [<ffffffff8025003c>] cpu_down+0x2b/0x42
[  488.264525]  [<ffffffff803b25b8>] store_online+0x27/0x71
[  488.264534]  [<ffffffff802b7b4a>] sysfs_write_file+0xb6/0xe5
[  488.264544]  [<ffffffff8027c6e5>] vfs_write+0xb2/0x155
[  488.264551]  [<ffffffff8027c83d>] sys_write+0x45/0x70
[  488.264560]  [<ffffffff80209a4e>] system_call+0x7e/0x83
[  488.264566] 
[  488.264570] 
[  488.264572] Code: 8b 75 00 4c 89 e7 e8 82 f8 ff ff f6 c7 02 c7 45 00 00 00 00 
[  488.264634] RIP  [<ffffffff8025d255>] __drain_pages+0x33/0x7d
[  488.264644]  RSP <ffff81016dd3fdf8>
[  488.264650] CR2: 0000000000000000
[  488.264655]  

Thanks,
Kiran

[-- Attachment #2: 2.6.19-rc3 --]
[-- Type: text/plain, Size: 31554 bytes --]

[    0.000000] Linux version 2.6.19-rc3 (kiran@x460) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.1)) #2 SMP PREEMPT Tue Oct 24 12:24:39 PDT 2006
[    0.000000] Command line: root=/dev/sda2 splash=silent showopts
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009b800 (usable)
[    0.000000]  BIOS-e820: 000000000009b800 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000d4000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 000000007ff10000 (usable)
[    0.000000]  BIOS-e820: 000000007ff10000 - 000000007ff19000 (ACPI data)
[    0.000000]  BIOS-e820: 000000007ff19000 - 000000007ff80000 (ACPI NVS)
[    0.000000]  BIOS-e820: 000000007ff80000 - 0000000080000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000e8000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000280000000 (usable)
[    0.000000] Entering add_active_range(0, 0, 155) 0 entries of 3200 used
[    0.000000] Entering add_active_range(0, 256, 524048) 1 entries of 3200 used
[    0.000000] Entering add_active_range(0, 1048576, 2621440) 2 entries of 3200 used
[    0.000000] end_pfn_map = 2621440
[    0.000000] DMI present.
[    0.000000] ACPI: RSDP (v000 PTLTD                                 ) @ 0x00000000000f7040
[    0.000000] ACPI: RSDT (v001 PTLTD    RSDT   0x06040000  LTP 0x00000000) @ 0x000000007ff153c7
[    0.000000] ACPI: FADT (v001 NVIDIA CK8S     0x06040000 PTL_ 0x000f4240) @ 0x000000007ff18d36
[    0.000000] ACPI: SRAT (v001 AMD    HAMMER   0x06040000 AMD  0x00000001) @ 0x000000007ff18daa
[    0.000000] ACPI: SPCR (v001 PTLTD  $UCRTBL$ 0x06040000 PTL  0x00000001) @ 0x000000007ff18eba
[    0.000000] ACPI: MCFG (v001 PTLTD    MCFG   0x06040000  LTP 0x00000000) @ 0x000000007ff18f0a
[    0.000000] ACPI: MADT (v001 PTLTD  	 APIC   0x06040000  LTP 0x00000000) @ 0x000000007ff18f46
[    0.000000] ACPI: BOOT (v001 PTLTD  $SBFTBL$ 0x06040000  LTP 0x00000001) @ 0x000000007ff18fd8
[    0.000000] ACPI: DSDT (v001 NVIDIA      CK8 0x06040000 MSFT 0x0100000e) @ 0x0000000000000000
[    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 1 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 2 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 3 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] Entering add_active_range(0, 0, 155) 0 entries of 3200 used
[    0.000000] SRAT: Node 0 PXM 0 0-80000000
[    0.000000] Entering add_active_range(0, 0, 155) 1 entries of 3200 used
[    0.000000] Entering add_active_range(0, 256, 524048) 1 entries of 3200 used
[    0.000000] SRAT: Node 0 PXM 0 0-180000000
[    0.000000] Entering add_active_range(0, 0, 155) 2 entries of 3200 used
[    0.000000] Entering add_active_range(0, 256, 524048) 2 entries of 3200 used
[    0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 3200 used
[    0.000000] SRAT: Node 1 PXM 1 180000000-280000000
[    0.000000] Entering add_active_range(1, 1572864, 2621440) 3 entries of 3200 used
[    0.000000] NUMA: Using 31 for the hash shift.
[    0.000000] Bootmem setup node 0 0000000000000000-0000000180000000
[    0.000000] Bootmem setup node 1 0000000180000000-0000000280000000
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA             0 ->     4096
[    0.000000]   DMA32        4096 ->  1048576
[    0.000000]   Normal    1048576 ->  2621440
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0:        0 ->      155
[    0.000000]     0:      256 ->   524048
[    0.000000]     0:  1048576 ->  1572864
[    0.000000]     1:  1572864 ->  2621440
[    0.000000] On node 0 totalpages: 1048235
[    0.000000]   DMA zone: 88 pages used for memmap
[    0.000000]   DMA zone: 2151 pages reserved
[    0.000000]   DMA zone: 1756 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 22440 pages used for memmap
[    0.000000]   DMA32 zone: 497512 pages, LIFO batch:31
[    0.000000]   Normal zone: 11264 pages used for memmap
[    0.000000]   Normal zone: 513024 pages, LIFO batch:31
[    0.000000] On node 1 totalpages: 1048576
[    0.000000]   DMA zone: 0 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages used for memmap
[    0.000000]   Normal zone: 22528 pages used for memmap
[    0.000000]   Normal zone: 1026048 pages, LIFO batch:31
[    0.000000] Nvidia board detected. Ignoring ACPI timer override.
[    0.000000] ACPI: PM-Timer IO Port: 0x8008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] Processor #0 (Bootup-CPU)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] Processor #1
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] Processor #2
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
[    0.000000] Processor #3
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: IOAPIC (id[0x05] address[0xd8000000] gsi_base[24])
[    0.000000] IOAPIC[1]: apic_id 5, address 0xd8000000, GSI 24-27
[    0.000000] ACPI: IOAPIC (id[0x06] address[0xd8001000] gsi_base[28])
[    0.000000] IOAPIC[2]: apic_id 6, address 0xd8001000, GSI 28-31
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Setting APIC routing to physical flat
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] Nosave address range: 000000000009b000 - 000000000009c000
[    0.000000] Nosave address range: 000000000009c000 - 00000000000a0000
[    0.000000] Nosave address range: 00000000000a0000 - 00000000000d4000
[    0.000000] Nosave address range: 00000000000d4000 - 0000000000100000
[    0.000000] Nosave address range: 000000007ff10000 - 000000007ff19000
[    0.000000] Nosave address range: 000000007ff19000 - 000000007ff80000
[    0.000000] Nosave address range: 000000007ff80000 - 0000000080000000
[    0.000000] Nosave address range: 0000000080000000 - 00000000e0000000
[    0.000000] Nosave address range: 00000000e0000000 - 00000000e8000000
[    0.000000] Nosave address range: 00000000e8000000 - 00000000fec00000
[    0.000000] Nosave address range: 00000000fec00000 - 00000000fee00000
[    0.000000] Nosave address range: 00000000fee00000 - 00000000fee01000
[    0.000000] Nosave address range: 00000000fee01000 - 00000000fff80000
[    0.000000] Nosave address range: 00000000fff80000 - 0000000100000000
[    0.000000] Allocating PCI resources starting at 88000000 (gap: 80000000:60000000)
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PERCPU: Allocating 37632 bytes of per cpu data
[    0.000000] Built 2 zonelists.  Total pages: 2038340
[    0.000000] Kernel command line: root=/dev/sda2 splash=silent showopts
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[  277.730450] Console: colour VGA+ 80x25
[  277.734534] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[  277.734605] ... MAX_LOCKDEP_SUBCLASSES:    8
[  277.734656] ... MAX_LOCK_DEPTH:          30
[  277.734706] ... MAX_LOCKDEP_KEYS:        2048
[  277.734758] ... CLASSHASH_SIZE:           1024
[  277.734809] ... MAX_LOCKDEP_ENTRIES:     8192
[  277.734860] ... MAX_LOCKDEP_CHAINS:      8192
[  277.734911] ... CHAINHASH_SIZE:          4096
[  277.734963]  memory used by lock dependency info: 1328 kB
[  277.735017]  per task-struct memory footprint: 1680 bytes
[  277.739261] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[  277.746508] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[  277.749122] Checking aperture...
[  277.749175] CPU 0: aperture @ 0 size 32 MB
[  277.751943] No AGP bridge found
[  277.751992] Your BIOS doesn't leave a aperture memory hole
[  277.752046] Please enable the IOMMU option in the BIOS setup
[  277.752099] This costs you 64 MB of RAM
[  277.785927] Mapping aperture over 65536 KB of RAM @ 4000000
[  277.900475] Memory: 8074684k/10485760k available (2735k kernel code, 312560k reserved, 1281k data, 304k init)
[  278.048243] Calibrating delay using timer specific routine.. 4824.91 BogoMIPS (lpj=24124594)
[  278.048925] Mount-cache hash table entries: 256
[  278.050110] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  278.050167] CPU: L2 Cache: 1024K (64 bytes/line)
[  278.050220] CPU 0/0 -> Node 0
[  278.050270] CPU: Physical Processor ID: 0
[  278.050320] CPU: Processor Core ID: 0
[  278.050389] lockdep: not fixing up alternatives.
[  278.050441] ACPI: Core revision 20060707
[  278.158599] Using local APIC timer interrupts.
[  278.200058] result 12557978
[  278.200107] Detected 12.557 MHz APIC timer.
[  278.209368] lockdep: not fixing up alternatives.
[  278.209740] Booting processor 1/4 APIC 0x1
[  278.220064] Initializing CPU#1
[  278.367716] Calibrating delay using timer specific routine.. 4822.38 BogoMIPS (lpj=24111902)
[  278.367722] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  278.367724] CPU: L2 Cache: 1024K (64 bytes/line)
[  278.367726] CPU 1/1 -> Node 0
[  278.367728] CPU: Physical Processor ID: 0
[  278.367730] CPU: Processor Core ID: 1
[  278.367848] Dual Core AMD Opteron(tm) Processor 280 stepping 02
[  278.377712] CPU 1: Syncing TSC to CPU 0.
[  278.377827] CPU 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 637 cycles)
[  278.378892] lockdep: not fixing up alternatives.
[  278.379570] Booting processor 2/4 APIC 0x2
[  278.389900] Initializing CPU#2
[  278.537439] Calibrating delay using timer specific routine.. 4822.38 BogoMIPS (lpj=24111923)
[  278.537446] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  278.537448] CPU: L2 Cache: 1024K (64 bytes/line)
[  278.537451] CPU 2/2 -> Node 1
[  278.537453] CPU: Physical Processor ID: 1
[  278.537454] CPU: Processor Core ID: 0
[  278.537568] Dual Core AMD Opteron(tm) Processor 280 stepping 02
[  278.547437] CPU 2: Syncing TSC to CPU 0.
[  278.547619] CPU 2: synchronized TSC with CPU 0 (last diff -17 cycles, maxerr 1238 cycles)
[  278.548616] lockdep: not fixing up alternatives.
[  278.549260] Booting processor 3/4 APIC 0x3
[  278.560124] Initializing CPU#3
[  278.707160] Calibrating delay using timer specific routine.. 4822.34 BogoMIPS (lpj=24111748)
[  278.707168] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  278.707170] CPU: L2 Cache: 1024K (64 bytes/line)
[  278.707172] CPU 3/3 -> Node 1
[  278.707175] CPU: Physical Processor ID: 1
[  278.707176] CPU: Processor Core ID: 1
[  278.707290] Dual Core AMD Opteron(tm) Processor 280 stepping 02
[  278.717159] CPU 3: Syncing TSC to CPU 0.
[  278.717342] CPU 3: synchronized TSC with CPU 0 (last diff -17 cycles, maxerr 1238 cycles)
[  278.717348] Brought up 4 CPUs
[  278.718069] testing NMI watchdog ... OK.
[  278.818025] Disabling vsyscall due to use of PM timer
[  278.818079] time.c: Using 3.579545 MHz WALL PM GTOD PM timer.
[  278.818133] time.c: Detected 2411.129 MHz processor.
[  279.037931] migration_cost=344,788
[  279.040588] NET: Registered protocol family 16
[  279.041267] ACPI: bus type pci registered
[  279.041322] PCI: Using configuration type 1
[  279.053074] ACPI: Interpreter enabled
[  279.053128] ACPI: Using IOAPIC for interrupt routing
[  279.055219] ACPI: PCI Root Bridge [PCI0] (0000:00)
[  279.055343] PCI: Probing PCI hardware (bus 00)
[  279.055723] ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
[  279.061084] Boot video device is 0000:01:07.0
[  279.061515] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[  279.109544] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P0._PRT]
[  279.112529] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.XVR0._PRT]
[  279.115015] ACPI: PCI Interrupt Link [LNK1] (IRQs 16 17 18 19) *0
[  279.116022] ACPI: PCI Interrupt Link [LNK2] (IRQs 16 17 18 19) *0
[  279.117040] ACPI: PCI Interrupt Link [LNK3] (IRQs 16 17 18 19) *0
[  279.118050] ACPI: PCI Interrupt Link [LNK4] (IRQs 16 17 18 19) *0, disabled.
[  279.119099] ACPI: PCI Interrupt Link [LNK5] (IRQs 16 17 18 19) *0, disabled.
[  279.120146] ACPI: PCI Interrupt Link [LSMB] (IRQs 20 21 22 23) *0, disabled.
[  279.121181] ACPI: PCI Interrupt Link [LUS0] (IRQs 20 21 22 23) *0
[  279.122193] ACPI: PCI Interrupt Link [LUS2] (IRQs 20 21 22 23) *0, disabled.
[  279.123240] ACPI: PCI Interrupt Link [LMAC] (IRQs 20 21 22 23) *0, disabled.
[  279.124280] ACPI: PCI Interrupt Link [LACI] (IRQs 20 21 22 23) *0, disabled.
[  279.125320] ACPI: PCI Interrupt Link [LMCI] (IRQs 20 21 22 23) *0, disabled.
[  279.126360] ACPI: PCI Interrupt Link [LPID] (IRQs 20 21 22 23) *0, disabled.
[  279.127429] ACPI: PCI Interrupt Link [LTID] (IRQs 20 21 22 23) *0
[  279.128458] ACPI: PCI Interrupt Link [LSI1] (IRQs 20 21 22 23) *0, disabled.
[  279.129520] ACPI: PCI Interrupt Link [APCP] (IRQs 20 21 22 23) *0, disabled.
[  279.130126] ACPI: PCI Root Bridge [PCI2] (0000:08)
[  279.130181] PCI: Probing PCI hardware (bus 08)
[  279.130330] ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
[  279.137089] ACPI: PCI Interrupt Routing Table [\_SB_.PCI2.G0PA._PRT]
[  279.137629] ACPI: PCI Interrupt Routing Table [\_SB_.PCI2.G0PB._PRT]
[  279.139419] SCSI subsystem initialized
[  279.139687] usbcore: registered new interface driver usbfs
[  279.139874] usbcore: registered new interface driver hub
[  279.141126] usbcore: registered new device driver usb
[  279.141275] PCI: Using ACPI for IRQ routing
[  279.141328] PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
[  279.141746] PCI-DMA: Disabling AGP.
[  279.141839] PCI-DMA: aperture base @ 4000000 size 65536 KB
[  279.141892] PCI-DMA: using GART IOMMU.
[  279.141945] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[  279.144501] PCI: Bridge: 0000:00:09.0
[  279.144553]   IO window: 2000-2fff
[  279.144605]   MEM window: c8100000-c9ffffff
[  279.144658]   PREFETCH window: 88000000-880fffff
[  279.144711] PCI: Bridge: 0000:00:0e.0
[  279.144762]   IO window: disabled.
[  279.144814]   MEM window: ca000000-ca0fffff
[  279.144867]   PREFETCH window: ca800000-d7ffffff
[  279.144925] PCI: Setting latency timer of device 0000:00:09.0 to 64
[  279.144934] PCI: Setting latency timer of device 0000:00:0e.0 to 64
[  279.144940] PCI: Bridge: 0000:08:0a.0
[  279.144991]   IO window: disabled.
[  279.145042]   MEM window: disabled.
[  279.145093]   PREFETCH window: disabled.
[  279.145149] PCI: Bridge: 0000:08:0b.0
[  279.145198]   IO window: disabled.
[  279.145251]   MEM window: d8100000-d81fffff
[  279.145304]   PREFETCH window: 88100000-881fffff
[  279.145897] NET: Registered protocol family 2
[  279.256530] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
[  279.258408] TCP established hash table entries: 65536 (order: 9, 3670016 bytes)
[  279.262505] TCP bind hash table entries: 32768 (order: 8, 1835008 bytes)
[  279.264617] TCP: Hash tables configured (established 65536 bind 32768)
[  279.264711] TCP reno registered
[  279.265327] Simple Boot Flag at 0x36 set to 0x1
[  279.266948] microcode: CPU0 not a capable Intel processor
[  279.267030] microcode: CPU1 not a capable Intel processor
[  279.267153] microcode: CPU2 not a capable Intel processor
[  279.267291] microcode: CPU3 not a capable Intel processor
[  279.267356] IA-32 Microcode Update Driver: v1.14a <tigran@veritas.com>
[  279.272881] SGI XFS with large block/inode numbers, no debug enabled
[  279.273920] io scheduler noop registered
[  279.274003] io scheduler anticipatory registered (default)
[  279.274118] io scheduler deadline registered
[  279.274284] io scheduler cfq registered
[  279.486101] PCI: Found HT MSI mapping on 0000:00:0e.0 with capability disabled
[  279.486186] PCI: Found HT MSI mapping on 0000:00:00.0 with capability enabled
[  279.486256] PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:08:0a.0 subordinate bus.
[  279.486333] PCI: MSI quirk detected. PCI_BUS_FLAGS_NO_MSI set for 0000:08:0b.0 subordinate bus.
[  279.557386] Real Time Clock Driver v1.12ac
[  279.557444] Linux agpgart interface v0.101 (c) Dave Jones
[  279.557500] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
[  279.558052] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[  279.558652] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[  279.561780] RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
[  279.562996] loop: loaded (max 8 devices)
[  279.563048] Intel(R) PRO/1000 Network Driver - version 7.2.9-k2
[  279.563103] Copyright (c) 1999-2006 Intel Corporation.
[  279.563396] tg3.c:v3.67 (October 18, 2006)
[  279.563470] ACPI: PCI Interrupt 0000:0a:09.0[A] -> GSI 28 (level, low) -> IRQ 28
[  279.593970] eth0: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:31:5f:ac
[  279.594232] eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[  279.594305] eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
[  279.594411] ACPI: PCI Interrupt 0000:0a:09.1[B] -> GSI 29 (level, low) -> IRQ 29
[  279.627921] eth1: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:31:5f:ad
[  279.628182] eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[  279.628255] eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
[  279.628843] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
[  279.628901] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
[  279.629036] NFORCE-CK804: IDE controller at PCI slot 0000:00:06.0
[  279.629106] NFORCE-CK804: chipset revision 242
[  279.629158] NFORCE-CK804: not 100% native mode: will probe irqs later
[  279.629217] NFORCE-CK804: 0000:00:06.0 (rev f2) UDMA133 controller
[  279.629277]     ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
[  279.629416]     ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
[  279.629552] Probing IDE interface ide0...
[  280.244638] Probing IDE interface ide1...
[  315.230991] ide1: Wait for ready failed before probe !
[  315.836959] libata version 2.00 loaded.
[  315.837066] sata_nv 0000:00:07.0: version 2.0
[  315.837804] ACPI: PCI Interrupt Link [LTID] enabled at IRQ 23
[  315.837867] ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [LTID] -> GSI 23 (level, high) -> IRQ 23
[  315.838403] PCI: Setting latency timer of device 0000:00:07.0 to 64
[  315.838733] ata1: SATA max UDMA/133 cmd 0x1440 ctl 0x1436 bmdma 0x1410 irq 23
[  315.838891] ata2: SATA max UDMA/133 cmd 0x1438 ctl 0x1432 bmdma 0x1418 irq 23
[  315.838965] scsi0 : sata_nv
[  316.175391] ata1: SATA link down (SStatus 0 SControl 300)
[  316.175469] scsi1 : sata_nv
[  316.504843] ata2: SATA link down (SStatus 0 SControl 300)
[  316.505656] ACPI: PCI Interrupt Link [LSI1] enabled at IRQ 22
[  316.505719] ACPI: PCI Interrupt 0000:00:08.0[A] -> Link [LSI1] -> GSI 22 (level, high) -> IRQ 22
[  316.506055] PCI: Setting latency timer of device 0000:00:08.0 to 64
[  316.506232] ata3: SATA max UDMA/133 cmd 0x1458 ctl 0x144E bmdma 0x1420 irq 22
[  316.506392] ata4: SATA max UDMA/133 cmd 0x1450 ctl 0x144A bmdma 0x1428 irq 22
[  316.506465] scsi2 : sata_nv
[  317.004028] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[  317.004687] ata3.00: ATA-7, max UDMA/133, 321672960 sectors: LBA48 NCQ (depth 0/1)
[  317.004759] ata3.00: ata3: dev 0 multi count 16
[  317.005487] ata3.00: configured for UDMA/133
[  317.005544] scsi3 : sata_nv
[  317.343453] ata4: SATA link down (SStatus 0 SControl 300)
[  317.344325] scsi 2:0:0:0: Direct-Access     ATA      WDC WD1600YD-01N 10.0 PQ: 0 ANSI: 5
[  317.345270] SCSI device sda: 321672960 512-byte hdwr sectors (164697 MB)
[  317.345344] sda: Write Protect is off
[  317.345395] sda: Mode Sense: 00 3a 00 00
[  317.345428] SCSI device sda: drive cache: write back
[  317.346721] SCSI device sda: 321672960 512-byte hdwr sectors (164697 MB)
[  317.346794] sda: Write Protect is off
[  317.346845] sda: Mode Sense: 00 3a 00 00
[  317.346878] SCSI device sda: drive cache: write back
[  317.347108]  sda: sda1 sda2 sda3
[  317.354142] sd 2:0:0:0: Attached scsi disk sda
[  317.354542] sd 2:0:0:0: Attached scsi generic sg0 type 0
[  317.354671] USB Universal Host Controller Interface driver v3.0
[  317.354895] Initializing USB Mass Storage driver...
[  317.355054] usbcore: registered new interface driver usb-storage
[  317.355111] USB Mass Storage support registered.
[  317.355268] usbcore: registered new interface driver usbhid
[  317.355324] drivers/usb/input/hid-core.c: v2.6:USB HID core driver
[  317.357957] serio: i8042 KBD port at 0x60,0x64 irq 1
[  317.358056] serio: i8042 AUX port at 0x60,0x64 irq 12
[  317.358437] mice: PS/2 mouse device common for all mice
[  317.358587] md: linear personality registered for level -1
[  317.358906] device-mapper: ioctl: 4.10.0-ioctl (2006-09-14) initialised: dm-devel@redhat.com
[  317.359082] TCP cubic registered
[  317.359168] NET: Registered protocol family 1
[  317.359233] NET: Registered protocol family 17
[  317.359702] ACPI: (supports S0 S1 S4 S5)
[  317.398060] input: AT Translated Set 2 keyboard as /class/input/input0
[  318.285432] input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
[  318.353007] md: Autodetecting RAID arrays.
[  318.353059] md: autorun ...
[  318.353107] md: ... autorun DONE.
[  318.361835] EXT3-fs: INFO: recovery required on readonly filesystem.
[  318.361891] EXT3-fs: write access will be enabled during recovery.
[  320.769049] kjournald starting.  Commit interval 5 seconds
[  320.769189] EXT3-fs: sda2: orphan cleanup on readonly fs
[  320.769252] ext3_orphan_cleanup: deleting unreferenced inode 1444609
[  320.770564] ext3_orphan_cleanup: deleting unreferenced inode 1444611
[  320.770582] ext3_orphan_cleanup: deleting unreferenced inode 2611585
[  320.770821] EXT3-fs: sda2: 3 orphan inodes deleted
[  320.770873] EXT3-fs: recovery complete.
[  320.778722] EXT3-fs: mounted filesystem with ordered data mode.
[  320.779037] VFS: Mounted root (ext3 filesystem) readonly.
[  320.779181] Freeing unused kernel memory: 304k freed
[  321.294430] warning: process `showconsole' used the removed sysctl system call
[  321.663630] warning: process `showconsole' used the removed sysctl system call
[  321.807213] EXT3 (no)acl options not supported
[  321.807271] EXT3 (no)user_xattr options not supported
[  322.029072] EXT3 (no)acl options not supported
[  322.029130] EXT3 (no)user_xattr options not supported
[  322.029572] EXT3 FS on sda2, internal journal
[  322.693490] warning: process `ls' used the removed sysctl system call
[  322.694815] warning: process `ls' used the removed sysctl system call
[  322.764619] warning: process `ls' used the removed sysctl system call
[  323.789944] EXT3 (no)acl options not supported
[  323.790001] EXT3 (no)user_xattr options not supported
[  323.794042] kjournald starting.  Commit interval 5 seconds
[  323.794279] EXT3 FS on sda1, internal journal
[  323.794365] EXT3-fs: mounted filesystem with ordered data mode.
[  324.195824] XFS mounting filesystem sda3
[  324.272445] Starting XFS recovery on filesystem: sda3 (logdev: internal)
[  324.427368] Ending XFS recovery on filesystem: sda3 (logdev: internal)
[  327.851923] PM: Writing back config space on device 0000:0a:09.0 at offset b (was 164814e4, writing 164414e4)
[  327.851937] PM: Writing back config space on device 0000:0a:09.0 at offset 3 (was 804000, writing 804010)
[  327.851943] PM: Writing back config space on device 0000:0a:09.0 at offset 2 (was 2000000, writing 2000003)
[  327.851948] PM: Writing back config space on device 0000:0a:09.0 at offset 1 (was 2b00000, writing 2b00106)
[  330.623384] tg3: eth0: Link is up at 1000 Mbps, full duplex.
[  330.623389] tg3: eth0: Flow control is on for TX and on for RX.
[  469.336719] 
[  469.336721] =============================================
[  469.336728] [ INFO: possible recursive locking detected ]
[  469.336731] 2.6.19-rc3 #2
[  469.336737] ---------------------------------------------
[  469.336745] rm/4538 is trying to acquire lock:
[  469.336753]  (&(&ip->i_lock)->mr_lock){----}, at: [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336782] 
[  469.336784] but task is already holding lock:
[  469.336791]  (&(&ip->i_lock)->mr_lock){----}, at: [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336816] 
[  469.336817] other info that might help us debug this:
[  469.336825] 3 locks held by rm/4538:
[  469.336832]  #0:  (&inode->i_mutex/1){--..}, at: [<ffffffff80286055>] do_unlinkat+0x75/0x136
[  469.336866]  #1:  (&inode->i_mutex){--..}, at: [<ffffffff80285f98>] vfs_unlink+0x45/0x8d
[  469.336897]  #2:  (&(&ip->i_lock)->mr_lock){----}, at: [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336928] 
[  469.336929] stack backtrace:
[  469.336936] 
[  469.336938] Call Trace:
[  469.336947]  [<ffffffff8024b4cc>] __lock_acquire+0x381/0xb8c
[  469.336957]  [<ffffffff8024bf80>] lock_acquire+0x4b/0x66
[  469.336966]  [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336976]  [<ffffffff80247c97>] down_write+0x1e/0x27
[  469.336985]  [<ffffffff80320050>] xfs_ilock+0x52/0x77
[  469.336993]  [<ffffffff8033cad1>] xfs_lock_dir_and_entry+0x92/0xce
[  469.337002]  [<ffffffff8033cd23>] xfs_remove+0x216/0x44a
[  469.337013]  [<ffffffff80346c96>] xfs_vn_unlink+0x21/0x4f
[  469.337024]  [<ffffffff804a72b9>] __mutex_lock_slowpath+0x249/0x279
[  469.337034]  [<ffffffff8024aaaf>] mark_held_locks+0x65/0x84
[  469.337043]  [<ffffffff804a72b9>] __mutex_lock_slowpath+0x249/0x279
[  469.337053]  [<ffffffff8024ac7c>] trace_hardirqs_on+0x101/0x128
[  469.337062]  [<ffffffff804a72dc>] __mutex_lock_slowpath+0x26c/0x279
[  469.337072]  [<ffffffff802832d9>] permission+0xa0/0xa9
[  469.337081]  [<ffffffff80285fb4>] vfs_unlink+0x61/0x8d
[  469.337090]  [<ffffffff80286098>] do_unlinkat+0xb8/0x136
[  469.337100]  [<ffffffff804a8649>] trace_hardirqs_on_thunk+0x35/0x37
[  469.337109]  [<ffffffff8024ac7c>] trace_hardirqs_on+0x101/0x128
[  469.337118]  [<ffffffff804a8649>] trace_hardirqs_on_thunk+0x35/0x37
[  469.337129]  [<ffffffff80209a4e>] system_call+0x7e/0x83
[  469.337137] 
[  487.063562] CPU 1 is now offline
[  487.134710] Breaking affinity for irq 22
[  487.243265] CPU 2 is now offline
[  487.422969] CPU 3 is now offline
[  487.422978] lockdep: not fixing up alternatives.
[  487.472345] lockdep: not fixing up alternatives.
[  487.472637] Booting processor 1/4 APIC 0x1
[  487.483163] Initializing CPU#1
[  487.625211] Calibrating delay using timer specific routine.. 4822.28 BogoMIPS (lpj=24111431)
[  487.625217] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  487.625219] CPU: L2 Cache: 1024K (64 bytes/line)
[  487.625221] CPU 1/1 -> Node 0
[  487.625223] CPU: Physical Processor ID: 0
[  487.625225] CPU: Processor Core ID: 1
[  487.625321] Dual Core AMD Opteron(tm) Processor 280 stepping 02
[  487.635210] CPU 1: Syncing TSC to CPU 0.
[  487.635306] CPU 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 638 cycles)
[  487.635487] microcode: CPU1 not a capable Intel processor
[  487.685423] lockdep: not fixing up alternatives.
[  487.685433] Booting processor 2/4 APIC 0x2
[  487.695710] Initializing CPU#2
[  487.844852] Calibrating delay using timer specific routine.. 4822.40 BogoMIPS (lpj=24112032)
[  487.844859] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  487.844861] CPU: L2 Cache: 1024K (64 bytes/line)
[  487.844864] CPU 2/2 -> Node 1
[  487.844866] CPU: Physical Processor ID: 1
[  487.844867] CPU: Processor Core ID: 0
[  487.844978] Dual Core AMD Opteron(tm) Processor 280 stepping 02
[  487.854844] CPU 2: Syncing TSC to CPU 0.
[  487.855026] CPU 2: synchronized TSC with CPU 0 (last diff -17 cycles, maxerr 1238 cycles)
[  487.855237] microcode: CPU2 not a capable Intel processor
[  487.904974] lockdep: not fixing up alternatives.
[  487.904984] Booting processor 3/4 APIC 0x3
[  487.915261] Initializing CPU#3
[  488.064492] Calibrating delay using timer specific routine.. 4822.42 BogoMIPS (lpj=24112120)
[  488.064500] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[  488.064502] CPU: L2 Cache: 1024K (64 bytes/line)
[  488.064504] CPU 3/3 -> Node 1
[  488.064507] CPU: Physical Processor ID: 1
[  488.064508] CPU: Processor Core ID: 1
[  488.064619] Dual Core AMD Opteron(tm) Processor 280 stepping 02
[  488.074489] CPU 3: Syncing TSC to CPU 0.
[  488.074673] CPU 3: synchronized TSC with CPU 0 (last diff -3 cycles, maxerr 1239 cycles)
[  488.074875] microcode: CPU3 not a capable Intel processor
[  488.264171] CPU 1 is now offline
[  488.264267] Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP: 
[  488.264271]  [<ffffffff8025d255>] __drain_pages+0x33/0x7d
[  488.264288] PGD 2753ae067 PUD 275f66067 PMD 0 
[  488.264302] Oops: 0000 [1] PREEMPT SMP 
[  488.264315] CPU 2 
[  488.264323] Modules linked in:
[  488.264332] Pid: 7617, comm: sh Not tainted 2.6.19-rc3 #2
[  488.264339] RIP: 0010:[<ffffffff8025d255>]  [<ffffffff8025d255>] __drain_pages+0x33/0x7d
[  488.264353] RSP: 0018:ffff81016dd3fdf8  EFLAGS: 00010046
[  488.264358] RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000000
[  488.264366] RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffff810180000000
[  488.264373] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000001
[  488.264381] R10: ffffffff8025cb12 R11: 0000000000000000 R12: ffff810180000000
[  488.264389] R13: 0000000000000001 R14: 0000000000000001 R15: ffff81027a7e5998
[  488.264397] FS:  00002af4d80a5ae0(0000) GS:ffff81018001ad18(0000) knlGS:0000000000000000
[  488.264405] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  488.264412] CR2: 0000000000000000 CR3: 00000002767bf000 CR4: 00000000000006a0
[  488.264420] Process sh (pid: 7617, threadinfo ffff81016dd3e000, task ffff81016e03d080)
[  488.264427] Stack:  0000000000000001 0000000000000001 0000000000000007 0000000000000001
[  488.264448]  000000000000000f ffffffff8025e771 ffffffff80538860 ffffffff8023e3c2
[  488.264466]  0000000000000001 0000000000000001 ffff81016e11a0c0 ffffffff8024ff50
[  488.264481] Call Trace:
[  488.264490]  [<ffffffff8025e771>] page_alloc_cpu_notify+0x18/0x33
[  488.264500]  [<ffffffff8023e3c2>] notifier_call_chain+0x23/0x32
[  488.264509]  [<ffffffff8024ff50>] _cpu_down+0x184/0x245
[  488.264516]  [<ffffffff8025003c>] cpu_down+0x2b/0x42
[  488.264525]  [<ffffffff803b25b8>] store_online+0x27/0x71
[  488.264534]  [<ffffffff802b7b4a>] sysfs_write_file+0xb6/0xe5
[  488.264544]  [<ffffffff8027c6e5>] vfs_write+0xb2/0x155
[  488.264551]  [<ffffffff8027c83d>] sys_write+0x45/0x70
[  488.264560]  [<ffffffff80209a4e>] system_call+0x7e/0x83
[  488.264566] 
[  488.264570] 
[  488.264572] Code: 8b 75 00 4c 89 e7 e8 82 f8 ff ff f6 c7 02 c7 45 00 00 00 00 
[  488.264634] RIP  [<ffffffff8025d255>] __drain_pages+0x33/0x7d
[  488.264644]  RSP <ffff81016dd3fdf8>
[  488.264650] CR2: 0000000000000000
[  488.264655]  

[-- Attachment #3: config.gz --]
[-- Type: application/x-gzip, Size: 6983 bytes --]