PROBLEM: oops

PROBLEM: oops

[Paweł Gołaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 37176 bytes --]


No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. Works for 
few minutes and crashes. Even SysRq doesn't work.

Dunno how to trigger it on other machine than mine (I have two, both 
affected). If more info is required - just ask.
[...]
[24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
[24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
[24483.211247] Oops: 0000 [#1] SMP 
[24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[24483.211878] 
[24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
[24483.211952] EIP: 0060:[<c081cf9a>] EFLAGS: 00010046 CPU: 2
[24483.211996] EIP is at pick_next_task_fair+0x6a/0x100
[24483.212032] EAX: c1d941e0 EBX: c1d9421c ECX: 00000000 EDX: 00000000
[24483.212073] ESI: 00000000 EDI: 00000000 EBP: c1d941e0 ESP: f71b7ebc
[24483.212115]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[24483.212153] Process md2_raid10 (pid: 1677, ti=f71b6000 task=f709c600 task.ti=f71b6000)
[24483.212196] Stack: 00000000 00000000 c1d941e0 c1d941e0 c0acbf00 c1d94690 00000000 c1d941e0 
[24483.212323]        c0a9cb29 f8a34b2c f72b1000 00000286 f8a35452 00000000 f709c600 c1d941e0 
[24483.212451]        c1d94690 00001644 00000002 c1d8f688 c1d8f684 00000000 000021e0 f709c794 
[24483.212577] Call Trace:
[24483.212626]  [<c0a9cb29>] schedule+0x5b9/0xc20
[24483.212697]  [<f8a34b2c>] cleanup_module+0x271c/0x47db [raid10]
[24483.212755]  [<f8a35452>] cleanup_module+0x3042/0x47db [raid10]
[24483.212805]  [<c0a9d50d>] schedule_timeout+0xbd/0xf0
[24483.212853]  [<c0804d53>] common_interrupt+0x23/0x30
[24483.212911]  [<f882d543>] md_register_thread+0x153/0x180 [md_mod]
[24483.212982]  [<c083bcd0>] autoremove_wake_function+0x0/0x40
[24483.213056]  [<f882d490>] md_register_thread+0xa0/0x180 [md_mod]
[24483.213111]  [<c083ba29>] kthread+0x39/0x70
[24483.213156]  [<c083b9f0>] kthread+0x0/0x70
[24483.213199]  [<c0805007>] kernel_thread_helper+0x7/0x10
[24483.213249]  =======================
[24483.213281] Code: 8b 4e 1c 85 c9 75 7f 8b 43 44 8b 90 18 04 00 00 8b 80 14 04 00 00 89 56 24 89 46 20 89 73 38 8b 46 28 8b 56 2c 89 46 38 89 56 3c <8b> 5e 58 85 db 74 67 8b 4b 28 31 f6 85 c9 74 f0 8b 73 44 8b 53 
[24483.213899] EIP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100 SS:ESP 0068:f71b7ebc
[24483.214257] ---[ end trace c9dd311f4b324acb ]---
[...]

There is some bugs catched too:
[  493.923295] BUG: soft lockup - CPU#0 stuck for 61s! [md2_raid10:1672]
[  493.923295] Modules linked in: softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc parport ehci_hcd floppy i2c_piix4 psmouse piix e100 thermal serio_raw e1000 container i2c_core mii usbcore pcspkr evdev ide_core sg button processor ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[  493.923295] 
[  493.923295] Pid: 1672, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
[  493.923295] EIP: 0060:[<f8a38ff5>] EFLAGS: 00010282 CPU: 0
[  493.923295] EIP is at cleanup_module+0x2be5/0x47db [raid10]
[  493.923295] EAX: f411c000 EBX: f2305c60 ECX: 000005bf EDX: 0000000d
[  493.923295] ESI: f411ca41 EDI: f1dffa41 EBP: 0000009c ESP: f71f3ef0
[  493.923295]  DS: 0068 ES: 007b FS: 00d8 GS: 0000 SS: 0068
[  493.923295] CR0: 8005003b CR2: 080bd054 CR3: 36461000 CR4: 000006d0
[  493.923295] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  493.923295] DR6: ffff0ff0 DR7: 00000400
[  493.923295]  [<c09e7c57>] ? net_rx_action+0x127/0x200
[  493.923295]  [<c0a9d50d>] ? schedule_timeout+0xbd/0xf0
[  493.923295]  [<c0804d53>] ? common_interrupt+0x23/0x30
[  493.923295]  [<c083be41>] ? finish_wait+0x31/0x80
[  493.923295]  [<f882d4b9>] ? md_register_thread+0xc9/0x180 [md_mod]
[  493.923295]  [<c083bcd0>] ? autoremove_wake_function+0x0/0x40
[  493.923295]  [<f882d490>] ? md_register_thread+0xa0/0x180 [md_mod]
[  493.923295]  [<c083ba29>] ? kthread+0x39/0x70
[  493.923295]  [<c083b9f0>] ? kthread+0x0/0x70
[  493.923295]  [<c0805007>] ? kernel_thread_helper+0x7/0x10
[  493.923295]  =======================



# cat /proc/mdstat
Personalities : [raid1] [raid10] 
md2 : active raid10 sda2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
      834929664 blocks 64K chunks 2 far-copies [6/6] [UUUUUU]
      [=>...................]  resync =  7.9% (66705088/834929664) finish=1816.4min speed=7046K/sec
      
md1 : active raid1 sde1[0] sdf1[2](S) sdd1[1]
      14651136 blocks [2/2] [UU]
      
md0 : active raid1 sda1[0] sdc1[2](S) sdb1[1]
      14651136 blocks [2/2] [UU]
      
unused devices: <none>

# mount
/dev/md0 on / type ext3 (rw)
none on /proc type proc (rw,gid=17)
usbfs on /proc/bus/usb type usbfs (rw,devgid=78,devmode=664)
sysfs on /sys type sysfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)
none on /dev/cpuset type cgroup (rw,cpuset,noprefix)
/dev/md1 on /vservers type ext3 (rw)
/dev/md2 on /vservers/news/srv type xfs (rw,noatime)
/vservers/news/srv/spool on /vservers/news/var/spool/news type none (rw,bind)
/vservers/news/srv/lib on /vservers/news/var/lib/news type none (rw,bind)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw,noexec,nosuid,nodev,mode=1777)

# cat /proc/version
Linux version 2.6.27.31-1 (blues@carme-pld-i686) (gcc version 4.4.1 20090724 (release) (PLD-Linux) ) #1 SMP Fri Aug 21 23:19:19 CEST 2009


4 processors (right now only one left for testing):
# cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 10
model name	: Pentium III (Cascades)
stepping	: 1
cpu MHz		: 700.184
cache size	: 1024 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips	: 1400.71
clflush size	: 32
power management:

# lspci 
00:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c810 (rev 23)
00:09.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 08)
00:0a.0 VGA compatible controller: Cirrus Logic GD 5480 (rev 23)
00:0b.0 PIC: Intel Corporation 683053 Programmable Interrupt Device
00:0c.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 02)
00:0c.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:0c.2 USB Controller: Intel Corporation 82371AB/EB/MB PIIX4 USB (rev 01)
00:0c.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
00:10.0 Host bridge: Intel Corporation 450NX - 82451NX Memory & I/O Controller (rev 03)
00:12.0 Host bridge: Intel Corporation 450NX - 82454NX/84460GX PCI Expander Bridge (rev 04)
00:13.0 Host bridge: Intel Corporation 450NX - 82454NX/84460GX PCI Expander Bridge (rev 04)
01:01.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller
01:02.0 SCSI storage controller: Adaptec AHA-3960D / AIC-7899A U160/m (rev 01)
01:02.1 SCSI storage controller: Adaptec AHA-3960D / AIC-7899A U160/m (rev 01)
01:03.0 SCSI storage controller: LSI Logic / Symbios Logic 53C896/897 (rev 07)
01:03.1 SCSI storage controller: LSI Logic / Symbios Logic 53C896/897 (rev 07)


# dmesg:
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.27.31-1 (blues@carme-pld-i686) (gcc version 4.4.1 20090724 (release) (PLD-Linux) ) #1 SMP Fri Aug 21 23:19:19 CEST 2009
[    0.000000] PAT WC disabled due to known CPU erratum.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009e000 (usable)
[    0.000000]  BIOS-e820: 000000000009e000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 000000005fffa800 (usable)
[    0.000000]  BIOS-e820: 000000005fffa800 - 000000005ffffc00 (ACPI data)
[    0.000000]  BIOS-e820: 000000005ffffc00 - 0000000060000000 (ACPI NVS)
[    0.000000] DMI 2.1 present.
[    0.000000] last_pfn = 0x5fffa max_arch_pfn = 0x100000
[    0.000000] kernel direct mapping tables up to 38000000 @ 7000-c000
[    0.000000] RAMDISK: 00e6c000 - 00eff13b
[    0.000000] 639MB HIGHMEM available.
[    0.000000] 896MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 38000000
[    0.000000]   low ram: 00000000 - 38000000
[    0.000000]   bootmap 00009000 - 00010000
[    0.000000] (9 early reservations) ==> bootmem [0000000000 - 0038000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0000001000 - 0000002000]    EX TRAMPOLINE ==> [0000001000 - 0000002000]
[    0.000000]   #2 [0000006000 - 0000007000]       TRAMPOLINE ==> [0000006000 - 0000007000]
[    0.000000]   #3 [0000200000 - 0000c8b120]    TEXT DATA BSS ==> [0000200000 - 0000c8b120]
[    0.000000]   #4 [0000e6c000 - 0000eff13b]          RAMDISK ==> [0000e6c000 - 0000eff13b]
[    0.000000]   #5 [0000c8c000 - 0000c91000]    INIT_PG_TABLE ==> [0000c8c000 - 0000c91000]
[    0.000000]   #6 [000009e000 - 0000100000]    BIOS reserved ==> [000009e000 - 0000100000]
[    0.000000]   #7 [0000007000 - 0000009000]          PGTABLE ==> [0000007000 - 0000009000]
[    0.000000]   #8 [0000009000 - 0000010000]          BOOTMAP ==> [0000009000 - 0000010000]
[    0.000000] found SMP MP-table at [c00f6b30] 000f6b30
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00038000
[    0.000000]   HighMem  0x00038000 -> 0x0005fffa
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x0000009e
[    0.000000]     0: 0x00000100 -> 0x0005fffa
[    0.000000] On node 0 totalpages: 393112
[    0.000000]   DMA zone: 3962 pages, LIFO batch:0
[    0.000000]   Normal zone: 223300 pages, LIFO batch:31
[    0.000000]   HighMem zone: 162394 pages, LIFO batch:31
[    0.000000] Intel MultiProcessor Specification v1.4
[    0.000000]     Virtual Wire compatibility mode.
[    0.000000] MPTABLE: OEM ID: INTEL   
[    0.000000] MPTABLE: Product ID: S450NX      
[    0.000000] MPTABLE: APIC at: 0xFEE00000
[    0.000000] Processor #3 (Bootup-CPU)
[    0.000000] Processor #0
[    0.000000] Processor #1
[    0.000000] Processor #2
[    0.000000] I/O APIC #4 Version 19 at 0xFEC00000.
[    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
[    0.000000] Processors: 4
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] mapped APIC to ffffb000 (fee00000)
[    0.000000] mapped IOAPIC to ffffa000 (fec00000)
[    0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
[    0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 70000000 (gap: 60000000:a0000000)
[    0.000000] PERCPU: Allocating 18108 bytes of per cpu data
[    0.000000] NR_CPUS: 32, nr_cpu_ids: 4, nr_node_ids 1
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 389656
[    0.000000] Kernel command line: BOOT_IMAGE=testing ro root=900 noapic nolapic acpi=off
[    0.000000] Enabling fast FPU save and restore... done.
[    0.000000] Enabling unmasked SIMD FPU exception support... done.
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 16384 bytes)
[    0.000000] TSC: Using PIT calibration value
[    0.000000] Detected 700.184 MHz processor.
[    0.003333] Console: colour VGA+ 80x25
[    0.003333] console [tty0] enabled
[    0.003333] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.003333] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.003333] Memory: 1545280k/1572840k available (2700k kernel code, 26172k reserved, 452k data, 6116k init, 655336k highmem)
[    0.003333] virtual kernel memory layout:
[    0.003333]     fixmap  : 0xffdf9000 - 0xfffff000   (2072 kB)
[    0.003333]     pkmap   : 0xff400000 - 0xff800000   (4096 kB)
[    0.003333]     vmalloc : 0xf8800000 - 0xff3fe000   ( 107 MB)
[    0.003333]     lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
[    0.003333]       .init : 0xc0207000 - 0xc0800000   (6116 kB)
[    0.003333]       .data : 0xc0ba2000 - 0xc0c133cc   ( 452 kB)
[    0.003333]       .text : 0xc0800000 - 0xc0aa3060   (2700 kB)
[    0.003333] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[    0.003333] CPA: page pool initialized 1 of 1 pages preallocated
[    0.003333] SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.003373] Calibrating delay loop (skipped), value calculated using timer frequency.. 1400.71 BogoMIPS (lpj=2333946)
[    0.003567] Security Framework initialized
[    0.003672] SELinux:  Disabled at boot.
[    0.003782] Mount-cache hash table entries: 512
[    0.004521] Initializing cgroup subsys ns
[    0.004593] Initializing cgroup subsys cpuacct
[    0.004655] Initializing cgroup subsys memory
[    0.004721] Initializing cgroup subsys devices
[    0.004836] CPU: L1 I cache: 16K, L1 D cache: 16K
[    0.004935] CPU: L2 cache: 1024K
[    0.004998] CPU serial number disabled.
[    0.005062] Intel machine check architecture supported.
[    0.005126] Intel machine check reporting enabled on CPU#0.
[    0.005227] Checking 'hlt' instruction... OK.
[    0.017889] BIOS bug, local APIC #3 not detected!...
[    0.017955] ... forcing use of dummy APIC emulation.(tell your hw vendor)
[    0.018018] SMP disabled
[    0.018309] Brought up 1 CPUs
[    0.018370] Total of 1 processors activated (1400.71 BogoMIPS).
[    0.019217] net_namespace: 840 bytes
[    0.019567] NET: Registered protocol family 16
[    0.020639] EISA bus registered
[    0.030160] bios32_service: base:000fd690 length:00000970 entry:000004b4
[    0.030240] PCI: PCI BIOS revision 2.10 entry at 0x4b4, last bus=1
[    0.030304] PCI: Using configuration type 1 for base access
[    0.033113] ACPI: Interpreter disabled.
[    0.033180] Linux Plug and Play Support v0.97 (c) Adam Belay
[    0.033368] pnp: PnP ACPI: disabled
[    0.033433] PnPBIOS: Scanning system for PnP BIOS support...
[    0.033525] PnPBIOS: Found PnP BIOS installation structure at 0xc00f6d90
[    0.033593] PnPBIOS: PnP BIOS version 1.0, entry 0xf0000:0xb2cb, dseg 0x400
[    0.039822] PnPBIOS: 20 nodes reported by PnP BIOS; 20 recorded by driver
[    0.040903] PCI: Probing PCI hardware
[    0.040968] PCI: Probing PCI hardware (bus 00)
[    0.041189] PCI: 0000:00:08.0 reg 10 io port: [2000, 20ff]
[    0.041204] PCI: 0000:00:08.0 reg 14 32bit mmio: [fa102000, fa1020ff]
[    0.041320] PCI: 0000:00:09.0 reg 10 32bit mmio: [fa100000, fa100fff]
[    0.041334] PCI: 0000:00:09.0 reg 14 io port: [2400, 243f]
[    0.041348] PCI: 0000:00:09.0 reg 18 32bit mmio: [fa000000, fa0fffff]
[    0.041379] PCI: 0000:00:09.0 reg 30 32bit mmio: [0, fffff]
[    0.041415] pci 0000:00:09.0: supports D1
[    0.041421] pci 0000:00:09.0: supports D2
[    0.041428] pci 0000:00:09.0: PME# supported from D0 D1 D2 D3hot
[    0.041497] pci 0000:00:09.0: PME# disabled
[    0.041581] PCI: 0000:00:0a.0 reg 10 32bit mmio: [fc000000, fdffffff]
[    0.041596] PCI: 0000:00:0a.0 reg 14 32bit mmio: [fa101000, fa101fff]
[    0.041633] PCI: 0000:00:0a.0 reg 30 32bit mmio: [0, 7fff]
[    0.041850] PCI: 0000:00:0c.1 reg 20 io port: [2460, 246f]
[    0.041939] PCI: 0000:00:0c.2 reg 20 io port: [2440, 245f]
[    0.041989] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
[    0.041993] * this clock source is slow. Consider trying other clock sources
[    0.042193] pci 0000:00:0c.3: quirk: region 0c00-0c3f claimed by PIIX4 ACPI
[    0.042263] pci 0000:00:0c.3: quirk: region 0cc0-0ccf claimed by PIIX4 SMB
[    0.042334] pci 0000:00:0c.3: PIIX4 devres B PIO at 0ca2-0ca3
[    0.042400] pci 0000:00:0c.3: PIIX4 devres C PIO at 0ca0-0ca1
[    0.042544] pci 0000:00:10.0: Searching for i450NX host bridges
[    0.042811] PCI: 0000:01:01.0 reg 10 32bit mmio: [fe040000, fe05ffff]
[    0.042826] PCI: 0000:01:01.0 reg 14 32bit mmio: [fe020000, fe03ffff]
[    0.042840] PCI: 0000:01:01.0 reg 18 io port: [3c00, 3c3f]
[    0.042872] PCI: 0000:01:01.0 reg 30 32bit mmio: [0, 1ffff]
[    0.042925] pci 0000:01:01.0: PME# supported from D0 D3hot D3cold
[    0.042993] pci 0000:01:01.0: PME# disabled
[    0.043093] PCI: 0000:01:02.0 reg 10 io port: [3000, 30ff]
[    0.043113] PCI: 0000:01:02.0 reg 14 64bit mmio: [fe004000, fe004fff]
[    0.043145] PCI: 0000:01:02.0 reg 30 32bit mmio: [0, 1ffff]
[    0.043219] PCI: 0000:01:02.1 reg 10 io port: [3400, 34ff]
[    0.043239] PCI: 0000:01:02.1 reg 14 64bit mmio: [fe005000, fe005fff]
[    0.043270] PCI: 0000:01:02.1 reg 30 32bit mmio: [0, 1ffff]
[    0.043454] PCI: 0000:01:03.0 reg 10 io port: [3800, 38ff]
[    0.043474] PCI: 0000:01:03.0 reg 14 64bit mmio: [fe006000, fe0063ff]
[    0.043494] PCI: 0000:01:03.0 reg 1c 64bit mmio: [fe000000, fe001fff]
[    0.043542] pci 0000:01:03.0: supports D1
[    0.043548] pci 0000:01:03.0: supports D2
[    0.043592] PCI: 0000:01:03.1 reg 10 io port: [4000, 40ff]
[    0.043612] PCI: 0000:01:03.1 reg 14 64bit mmio: [fe006400, fe0067ff]
[    0.043631] PCI: 0000:01:03.1 reg 1c 64bit mmio: [fe002000, fe003fff]
[    0.043679] pci 0000:01:03.1: supports D1
[    0.043684] pci 0000:01:03.1: supports D2
[    0.047163] NetLabel: Initializing
[    0.047163] NetLabel:  domain hash size = 128
[    0.047163] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.047163] NetLabel:  unlabeled traffic allowed by default
[    0.047163] system 00:00: ioport range 0xca0-0xca3 has been reserved
[    0.047208] system 00:02: iomem range 0xffffffffffe00000-0xffffffffffffffff has been reserved
[    0.047314] system 00:04: iomem range 0xdc000-0xdffff has been reserved
[    0.047394] system 00:06: iomem range 0x0-0x9ffff could not be reserved
[    0.047462] system 00:06: iomem range 0xe4000-0xfffff could not be reserved
[    0.047531] system 00:06: iomem range 0x100000-0x5fffffff could not be reserved
[    0.047632] system 00:0f: ioport range 0x4d0-0x4d1 has been reserved
[    0.047701] system 00:0f: ioport range 0xc00-0xc3f has been reserved
[    0.047767] system 00:0f: ioport range 0xcc0-0xccf has been reserved
[    0.047844] system 00:10: iomem range 0xfffffffffec00000-0xfffffffffec0ffff has been reserved
[    0.047926] system 00:10: iomem range 0xfffffffffee00000-0xfffffffffee00fff has been reserved
[    0.048020] system 00:12: iomem range 0xcfe00-0xcffff has been reserved
[    0.050358] bus: 00 index 0 io port: [0, ffff]
[    0.050420] bus: 00 index 1 mmio: [0, ffffffffffffffff]
[    0.050481] bus: 01 index 0 io port: [0, ffff]
[    0.050541] bus: 01 index 1 mmio: [0, ffffffffffffffff]
[    0.050603] bus: 80 index 0 io port: [0, ffff]
[    0.050664] bus: 80 index 1 mmio: [0, ffffffffffffffff]
[    0.050747] NET: Registered protocol family 2
[    0.051061] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.051597] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.057027] TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
[    0.060455] TCP: Hash tables configured (established 131072 bind 65536)
[    0.060541] TCP reno registered
[    0.060986] NET: Registered protocol family 1
[    0.061484] checking if image is initramfs...it isn't (no cpio magic); looks like an initrd
[    0.138243] Freeing initrd memory: 588k freed
[    0.140179] audit: initializing netlink socket (disabled)
[    0.140301] type=2000 audit(1251202105.139:1): initialized
[    0.162728] highmem bounce pool size: 64 pages
[    0.162812] HugeTLB registered 4 MB page size, pre-allocated 0 pages
[    0.172067] VFS: Disk quotas dquot_6.5.1
[    0.173658] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.174220] msgmni has been set to 1741
[    0.174718] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.174799] io scheduler noop registered
[    0.174859] io scheduler anticipatory registered
[    0.174920] io scheduler deadline registered
[    0.175022] io scheduler cfq registered (default)
[    0.175179] pci 0000:00:09.0: Firmware left e100 interrupts enabled; disabling
[    0.175272] pci 0000:00:0a.0: Boot video device
[    0.176285] isapnp: Scanning for PnP cards...
[    0.533412] isapnp: No Plug & Play device found
[    0.653627] Serial: 8250/16550 driver4 ports, IRQ sharing enabled
[    0.906736] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    1.160067] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.161903] 00:13: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    1.162739] 00:19: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.168033] brd: module loaded
[    1.168336] input: Macintosh mouse button emulation as /devices/virtual/input/input0
[    1.168736] PNP: PS/2 Controller [PNP0303] at 0x60,0x64 irq 1
[    1.168803] PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
[    1.169661] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.170681] mice: PS/2 mouse device common for all mice
[    1.171175] rtc_cmos 00:0a: rtc core: registered rtc_cmos as rtc0
[    1.171268] rtc0: alarms up to one day
[    1.171674] EISA: Probing bus 0 at eisa.0
[    1.171757] Cannot allocate resource for EISA slot 2
[    1.171820] Cannot allocate resource for EISA slot 3
[    1.171883] Cannot allocate resource for EISA slot 4
[    1.171970] EISA: Detected 0 cards.
[    1.172529] TCP cubic registered
[    1.173312] NET: Registered protocol family 10
[    1.173840] lo: Disabled Privacy Extensions
[    1.174004] Mobile IPv6
[    1.174067] NET: Registered protocol family 17
[    1.174156] Using IPI No-Shortcut mode
[    1.174791] TuxOnIce: Ignoring late initcall, as requested.
[    1.174881] registered taskstats version 1
[    1.175464] rtc_cmos 00:0a: setting system clock to 2009-08-25 12:08:27 UTC (1251202107)
[    1.194748] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    1.276725] RAMDISK: Compressed image found at block 0
[    1.363099] VFS: Mounted root (romfs filesystem) readonly.
[    1.407595] md: raid1 personality registered for level 1
[    1.467379] SCSI subsystem initialized
[    1.493361] sym0: <896> rev 0x7 at pci 0000:01:03.0 irq 11
[    1.495546] sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
[    1.496563] sym0: SCSI BUS has been reset.
[    1.502316] scsi0 : sym-2.2.3
[    1.504139] sym1: <896> rev 0x7 at pci 0000:01:03.1 irq 11
[    1.506293] sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking
[    1.506561] sym1: SCSI BUS has been reset.
[    1.513030] scsi1 : sym-2.2.3
[    1.514852] sym2: <810a> rev 0x23 at pci 0000:00:08.0 irq 11
[    1.517008] sym2: No NVRAM, ID 7, Fast-10, SE, parity checking
[    1.519894] sym2: SCSI BUS has been reset.
[    1.523511] scsi2 : sym-2.2.3
[   16.752206] scsi3 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
[   16.752212]         <Adaptec 3960D Ultra160 SCSI adapter>
[   16.752216]         aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
[   16.752220] 
[   16.964142] scsi 3:0:0:0: Direct-Access     FUJITSU  MBA3300NC        0102 PQ: 0 ANSI: 3
[   16.964240] scsi3:A:0:0: Tagged Queuing enabled.  Depth 253
[   16.964363] scsi target3:0:0: Beginning Domain Validation
[   16.968740] scsi target3:0:0: wide asynchronous
[   16.972171] scsi target3:0:0: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   16.979661] scsi target3:0:0: Ending Domain Validation
[   16.981185] scsi 3:0:1:0: Direct-Access     FUJITSU  MBA3300NC        0102 PQ: 0 ANSI: 3
[   16.981272] scsi3:A:1:0: Tagged Queuing enabled.  Depth 253
[   16.981384] scsi target3:0:1: Beginning Domain Validation
[   16.985757] scsi target3:0:1: wide asynchronous
[   16.989191] scsi target3:0:1: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   16.996635] scsi target3:0:1: Ending Domain Validation
[   16.998161] scsi 3:0:2:0: Direct-Access     FUJITSU  MBA3300NC        0102 PQ: 0 ANSI: 3
[   16.998246] scsi3:A:2:0: Tagged Queuing enabled.  Depth 253
[   16.998356] scsi target3:0:2: Beginning Domain Validation
[   17.002725] scsi target3:0:2: wide asynchronous
[   17.006152] scsi target3:0:2: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   17.013644] scsi target3:0:2: Ending Domain Validation
[   17.015195] scsi 3:0:3:0: Direct-Access     FUJITSU  MBA3300NC        0102 PQ: 0 ANSI: 3
[   17.015282] scsi3:A:3:0: Tagged Queuing enabled.  Depth 253
[   17.015394] scsi target3:0:3: Beginning Domain Validation
[   17.019758] scsi target3:0:3: wide asynchronous
[   17.023185] scsi target3:0:3: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   17.030671] scsi target3:0:3: Ending Domain Validation
[   17.032221] scsi 3:0:4:0: Direct-Access     FUJITSU  MBA3300NC        0102 PQ: 0 ANSI: 3
[   17.032309] scsi3:A:4:0: Tagged Queuing enabled.  Depth 253
[   17.032422] scsi target3:0:4: Beginning Domain Validation
[   17.036780] scsi target3:0:4: wide asynchronous
[   17.040196] scsi target3:0:4: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   17.047697] scsi target3:0:4: Ending Domain Validation
[   17.049225] scsi 3:0:5:0: Direct-Access     FUJITSU  MBA3300NC        0102 PQ: 0 ANSI: 3
[   17.049314] scsi3:A:5:0: Tagged Queuing enabled.  Depth 253
[   17.049429] scsi target3:0:5: Beginning Domain Validation
[   17.053778] scsi target3:0:5: wide asynchronous
[   17.057189] scsi target3:0:5: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 127)
[   17.064676] scsi target3:0:5: Ending Domain Validation
[   17.126291] scsi 3:0:6:0: Processor         ESG-SHV  SCA HSBP M6      0.65 PQ: 0 ANSI: 2
[   17.126386] scsi target3:0:6: Beginning Domain Validation
[   17.258886] scsi target3:0:6: Ending Domain Validation
[   31.964495] scsi4 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
[   31.964500]         <Adaptec 3960D Ultra160 SCSI adapter>
[   31.964504]         aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
[   31.964508] 
[   32.043887] libata version 3.00 loaded.
[   32.055564] ata_piix 0000:00:0c.1: version 2.12
[   32.055830] scsi5 : ata_piix
[   32.056297] scsi6 : ata_piix
[   32.056631] ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0x2460 irq 14
[   32.056700] ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0x2468 irq 15
[   32.261229] ata2: port disabled. ignoring.
[   32.261342] ata_piix 0000:00:10.0: no available native port
[   32.277970] Driver 'sd' needs updating - please use bus_type methods
[   32.279198] sd 3:0:0:0: [sda] 585937500 512-byte hardware sectors (300000 MB)
[   32.282022] sd 3:0:0:0: [sda] Write Protect is off
[   32.282086] sd 3:0:0:0: [sda] Mode Sense: b3 00 00 08
[   32.283374] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.284064] sd 3:0:0:0: [sda] 585937500 512-byte hardware sectors (300000 MB)
[   32.286875] sd 3:0:0:0: [sda] Write Protect is off
[   32.286939] sd 3:0:0:0: [sda] Mode Sense: b3 00 00 08
[   32.288223] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.288309]  sda: sda1 sda2
[   32.291443] sd 3:0:0:0: [sda] Attached SCSI disk
[   32.292539] sd 3:0:1:0: [sdb] 585937500 512-byte hardware sectors (300000 MB)
[   32.295345] sd 3:0:1:0: [sdb] Write Protect is off
[   32.295409] sd 3:0:1:0: [sdb] Mode Sense: b3 00 00 08
[   32.296692] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.297342] sd 3:0:1:0: [sdb] 585937500 512-byte hardware sectors (300000 MB)
[   32.300151] sd 3:0:1:0: [sdb] Write Protect is off
[   32.300217] sd 3:0:1:0: [sdb] Mode Sense: b3 00 00 08
[   32.301490] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.301575]  sdb: sdb1 sdb2
[   32.304018] sd 3:0:1:0: [sdb] Attached SCSI disk
[   32.305119] sd 3:0:2:0: [sdc] 585937500 512-byte hardware sectors (300000 MB)
[   32.307948] sd 3:0:2:0: [sdc] Write Protect is off
[   32.308013] sd 3:0:2:0: [sdc] Mode Sense: b3 00 00 08
[   32.309297] sd 3:0:2:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.309940] sd 3:0:2:0: [sdc] 585937500 512-byte hardware sectors (300000 MB)
[   32.312733] sd 3:0:2:0: [sdc] Write Protect is off
[   32.312799] sd 3:0:2:0: [sdc] Mode Sense: b3 00 00 08
[   32.314130] sd 3:0:2:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.314214]  sdc: sdc1 sdc2
[   32.318452] sd 3:0:2:0: [sdc] Attached SCSI disk
[   32.319538] sd 3:0:3:0: [sdd] 585937500 512-byte hardware sectors (300000 MB)
[   32.322352] sd 3:0:3:0: [sdd] Write Protect is off
[   32.322416] sd 3:0:3:0: [sdd] Mode Sense: b3 00 00 08
[   32.323699] sd 3:0:3:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.324341] sd 3:0:3:0: [sdd] 585937500 512-byte hardware sectors (300000 MB)
[   32.327152] sd 3:0:3:0: [sdd] Write Protect is off
[   32.327217] sd 3:0:3:0: [sdd] Mode Sense: b3 00 00 08
[   32.328498] sd 3:0:3:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.328585]  sdd: sdd1 sdd2
[   32.332610] sd 3:0:3:0: [sdd] Attached SCSI disk
[   32.333711] sd 3:0:4:0: [sde] 585937500 512-byte hardware sectors (300000 MB)
[   32.336513] sd 3:0:4:0: [sde] Write Protect is off
[   32.336578] sd 3:0:4:0: [sde] Mode Sense: b3 00 00 08
[   32.337864] sd 3:0:4:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.338504] sd 3:0:4:0: [sde] 585937500 512-byte hardware sectors (300000 MB)
[   32.341317] sd 3:0:4:0: [sde] Write Protect is off
[   32.341381] sd 3:0:4:0: [sde] Mode Sense: b3 00 00 08
[   32.342664] sd 3:0:4:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.342750]  sde: sde1 sde2
[   32.347933] sd 3:0:4:0: [sde] Attached SCSI disk
[   32.349027] sd 3:0:5:0: [sdf] 585937500 512-byte hardware sectors (300000 MB)
[   32.351825] sd 3:0:5:0: [sdf] Write Protect is off
[   32.351890] sd 3:0:5:0: [sdf] Mode Sense: b3 00 00 08
[   32.353169] sd 3:0:5:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.355024] sd 3:0:5:0: [sdf] 585937500 512-byte hardware sectors (300000 MB)
[   32.357821] sd 3:0:5:0: [sdf] Write Protect is off
[   32.357885] sd 3:0:5:0: [sdf] Mode Sense: b3 00 00 08
[   32.359168] sd 3:0:5:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   32.359252]  sdf: sdf1 sdf2
[   32.365039] sd 3:0:5:0: [sdf] Attached SCSI disk
[   32.366920] scsi: waiting for bus probes to complete ...
[   35.867985] md: md0 stopped.
[   35.892652] md: bind<sdb1>
[   35.893147] md: bind<sdc1>
[   35.893479] md: bind<sda1>
[   35.893615] md: md0: raid array is not clean -- starting background reconstruction
[   35.893988] raid1: raid set md0 active with 2 out of 2 mirrors
[   35.895418] md: resync of RAID array md0
[   35.895482] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[   35.895546] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[   35.895631] md: using 128k window, over a total of 14651136 blocks.
[   35.935801] EXT3-fs: INFO: recovery required on readonly filesystem.
[   35.935873] EXT3-fs: write access will be enabled during recovery.
[   36.115774] kjournald starting.  Commit interval 5 seconds
[   36.115888] EXT3-fs: recovery complete.
[   36.116848] EXT3-fs: mounted filesystem with ordered data mode.
[   36.116967] VFS: Mounted root (ext3 filesystem) readonly.
[   36.117031] Trying to move old root to /initrd ... okay
[   36.157621] Freeing unused kernel memory: 6116k freed
[   37.608836] Uniform Multi-Platform E-IDE driver
[   37.621007] Intel(R) PRO/1000 Network Driver - version 7.3.21-k3-NAPI
[   37.621016] Copyright (c) 1999-2006 Intel Corporation.
[   37.935157] e1000: 0000:01:01.0: e1000_probe: (PCI:33MHz:32-bit) 00:0e:0c:6f:8e:9e
[   38.175107] usbcore: registered new interface driver usbfs
[   38.175214] usbcore: registered new interface driver hub
[   38.193017] sd 3:0:0:0: Attached scsi generic sg0 type 0
[   38.193165] sd 3:0:1:0: Attached scsi generic sg1 type 0
[   38.193307] sd 3:0:2:0: Attached scsi generic sg2 type 0
[   38.193449] sd 3:0:3:0: Attached scsi generic sg3 type 0
[   38.193593] sd 3:0:4:0: Attached scsi generic sg4 type 0
[   38.193736] sd 3:0:5:0: Attached scsi generic sg5 type 0
[   38.193887] scsi 3:0:6:0: Attached scsi generic sg6 type 3
[   38.235471] usbcore: registered new device driver usb
[   38.269159] input: PC Speaker as /devices/platform/pcspkr/input/input2
[   38.280060] e100: Intel(R) PRO/100 Network Driver, 3.5.23-k4-NAPI
[   38.280069] e100: Copyright(c) 1999-2006 Intel Corporation
[   38.296991] piix: 450NX errata present, disabling IDE DMA.
[   38.297003] piix: A BIOS update may resolve this.
[   38.459851] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[   38.483980] e100 0000:00:09.0: PME# disabled
[   38.657629] e100: eth1: e100_probe: addr 0xfa100000, irq 10, MAC addr 00:02:b3:08:a0:68
[   38.657895] piix4_smbus 0000:00:0c.3: SMBus Host Controller at 0xcc0, revision 0
[   38.658035] piix 0000:00:10.0: IDE controller (0x8086:0x84ca rev 0x03)
[   38.658061] piix 0000:00:10.0: device not capable of full native PCI mode
[   38.658069] piix 0000:00:10.0: device disabled (BIOS)
[   38.734420] eepro100.c:v1.09j-t 9/29/99 Donald Becker
[   38.734425] eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <saw@saw.sw.com.sg> and others
[   38.742347] USB Universal Host Controller Interface driver v3.0
[   38.742551] uhci_hcd 0000:00:0c.2: UHCI Host Controller
[   38.742691] uhci_hcd 0000:00:0c.2: new USB bus registered, assigned bus number 1
[   38.742743] uhci_hcd 0000:00:0c.2: irq 15, io base 0x00002440
[   38.743393] usb usb1: configuration #1 chosen from 1 choice
[   38.743503] hub 1-0:1.0: USB hub found
[   38.743526] hub 1-0:1.0: 2 ports detected
[   38.844675] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[   38.844688] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   38.844696] usb usb1: Product: UHCI Host Controller
[   38.844703] usb usb1: Manufacturer: Linux 2.6.27.31-1 uhci_hcd
[   38.844709] usb usb1: SerialNumber: 0000:00:0c.2
[   39.806488] EXT3 FS on md0, internal journal
[   40.243241] md: md1 stopped.
[   40.303699] md: bind<sdd1>
[   40.304014] md: bind<sdf1>
[   40.304300] md: bind<sde1>
[   40.311908] raid1: raid set md1 active with 2 out of 2 mirrors
[   40.315430] md: md2 stopped.
[   40.371125] md: bind<sdb2>
[   40.372718] md: bind<sdc2>
[   40.373983] md: bind<sdd2>
[   40.374305] md: bind<sde2>
[   40.375607] md: bind<sdf2>
[   40.375974] md: bind<sda2>
[   40.376100] md: md2: raid array is not clean -- starting background reconstruction
[   40.394284] md: raid10 personality registered for level 10
[   40.395455] raid10: raid set md2 active with 6 out of 6 devices
[   40.409200] md: delaying resync of md2 until md0 has finished (they share one or more physical units)
[   40.710194] kjournald starting.  Commit interval 5 seconds
[   40.714142] EXT3 FS on md1, internal journal
[   40.714158] EXT3-fs: mounted filesystem with ordered data mode.
[   40.799280] SGI XFS with ACLs, security attributes, large block numbers, no debug enabled
[   40.803497] SGI XFS Quota Management subsystem
[   40.804921] Filesystem "md2": Disabling barriers, trial barrier write failed
[   40.809749] XFS mounting filesystem md2
[   40.963163] Starting XFS recovery on filesystem: md2 (logdev: internal)
[   41.181795] Ending XFS recovery on filesystem: md2 (logdev: internal)
[   42.252799] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   42.275605] e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   42.276359] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   52.217084] Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec (nowayout= 0)
[   52.406426] eth0: no IPv6 routers present
[   95.706099] scdp uses obsolete (PF_INET,SOCK_PACKET)
[  372.255222] md: md0: resync done.
[  372.285695] md: resync of RAID array md2
[  372.285704] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  372.285712] md: using maximum available idle IO bandwidth (but not more than 15000 KB/sec) for resync.
[  372.285727] md: using 128k window, over a total of 834929664 blocks.
[  372.409504] RAID1 conf printout:
[  372.409522]  --- wd:2 rd:2
[  372.409532]  disk 0, wo:0, o:1, dev:sda1
[  372.409540]  disk 1, wo:0, o:1, dev:sdb1


-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Tue, 2009-08-25 at 15:55 +0200, Paweł Gołaszewski wrote:
> No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. Works for 
> few minutes and crashes. Even SysRq doesn't work.
> 
> Dunno how to trigger it on other machine than mine (I have two, both 
> affected). If more info is required - just ask.
> [...]
> [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> [24483.211247] Oops: 0000 [#1] SMP 
> [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> [24483.211878] 
> [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)

So you're saying .30 crashes at the exact same place?

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1460 bytes --]

On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. Works for 
> > few minutes and crashes. Even SysRq doesn't work.
> > 
> > Dunno how to trigger it on other machine than mine (I have two, both 
> > affected). If more info is required - just ask.
> > [...]
> > [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> > [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> > [24483.211247] Oops: 0000 [#1] SMP 
> > [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> > [24483.211878] 
> > [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
> 
> So you're saying .30 crashes at the exact same place?

Well... I'm not able to catch anything on .30
No sign on netconsole, black screen on monitor...

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Tue, 2009-08-25 at 16:11 +0200, Pawel Golaszewski wrote:
> On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. Works for 
> > > few minutes and crashes. Even SysRq doesn't work.
> > > 
> > > Dunno how to trigger it on other machine than mine (I have two, both 
> > > affected). If more info is required - just ask.
> > > [...]
> > > [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> > > [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> > > [24483.211247] Oops: 0000 [#1] SMP 
> > > [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> > > [24483.211878] 
> > > [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
> > 
> > So you're saying .30 crashes at the exact same place?
> 
> Well... I'm not able to catch anything on .30
> No sign on netconsole, black screen on monitor...

Could you share your .config?

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1616 bytes --]

On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > > No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. 
> > > > Works for few minutes and crashes. Even SysRq doesn't work. Dunno 
> > > > how to trigger it on other machine than mine (I have two, both 
> > > > affected). If more info is required - just ask.
> > > > [...]
> > > > [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> > > > [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> > > > [24483.211247] Oops: 0000 [#1] SMP 
> > > > [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> > > > [24483.211878] 
> > > > [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
> > > 
> > > So you're saying .30 crashes at the exact same place?
> > 
> > Well... I'm not able to catch anything on .30 No sign on netconsole, 
> > black screen on monitor...
> 
> Could you share your .config?

Sure, I forgot:
http://www.blues.gda.pl/2.6.27.31.config.bz2

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Tue, 2009-08-25 at 16:28 +0200, Pawel Golaszewski wrote:
> On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > > > No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. 
> > > > > Works for few minutes and crashes. Even SysRq doesn't work. Dunno 
> > > > > how to trigger it on other machine than mine (I have two, both 
> > > > > affected). If more info is required - just ask.
> > > > > [...]
> > > > > [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> > > > > [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> > > > > [24483.211247] Oops: 0000 [#1] SMP 
> > > > > [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> > > > > [24483.211878] 
> > > > > [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
> > > > 
> > > > So you're saying .30 crashes at the exact same place?
> > > 
> > > Well... I'm not able to catch anything on .30 No sign on netconsole, 
> > > black screen on monitor...
> > 
> > Could you share your .config?
> 
> Sure, I forgot:
> http://www.blues.gda.pl/2.6.27.31.config.bz2

Right, are you actually using group scheduling? could you try to
reproduce without that?

CONFIG_GROUP_SCHED=n

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1878 bytes --]

On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > > > > No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. 
> > > > > > Works for few minutes and crashes. Even SysRq doesn't work. Dunno 
> > > > > > how to trigger it on other machine than mine (I have two, both 
> > > > > > affected). If more info is required - just ask.
> > > > > > [...]
> > > > > > [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> > > > > > [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> > > > > > [24483.211247] Oops: 0000 [#1] SMP 
> > > > > > [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> > > > > > [24483.211878] 
> > > > > > [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
> > > > > So you're saying .30 crashes at the exact same place?
> > > > Well... I'm not able to catch anything on .30 No sign on 
> > > > netconsole, black screen on monitor...
> > > Could you share your .config?
> > Sure, I forgot: http://www.blues.gda.pl/2.6.27.31.config.bz2
> Right, are you actually using group scheduling?

Not, yet - I wanted to start using it...

> could you try to reproduce without that?
> 
> CONFIG_GROUP_SCHED=n

I'll try. 

Do you think it's source of a problem?

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Tue, 2009-08-25 at 17:03 +0200, Pawel Golaszewski wrote:
> On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > > > > > No matter which kernel I'm using (2.6.27 - 2.6.30) it crashes. 
> > > > > > > Works for few minutes and crashes. Even SysRq doesn't work. Dunno 
> > > > > > > how to trigger it on other machine than mine (I have two, both 
> > > > > > > affected). If more info is required - just ask.
> > > > > > > [...]
> > > > > > > [24483.211036] BUG: unable to handle kernel NULL pointer dereference at 00000058
> > > > > > > [24483.211144] IP: [<c081cf9a>] pick_next_task_fair+0x6a/0x100
> > > > > > > [24483.211247] Oops: 0000 [#1] SMP 
> > > > > > > [24483.211300] Modules linked in: netconsole configfs dm_mod softdog sch_sfq xfs raid10 uhci_hcd eepro100 ppdev parport_pc floppy parport ehci_hcd e100 piix i2c_piix4 thermal psmouse e1000 container ide_core usbcore mii button pcspkr serio_raw processor i2c_core sg evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
> > > > > > > [24483.211878] 
> > > > > > > [24483.211910] Pid: 1677, comm: md2_raid10 Not tainted (2.6.27.31-1 #1)
> > > > > > So you're saying .30 crashes at the exact same place?
> > > > > Well... I'm not able to catch anything on .30 No sign on 
> > > > > netconsole, black screen on monitor...
> > > > Could you share your .config?
> > > Sure, I forgot: http://www.blues.gda.pl/2.6.27.31.config.bz2
> > Right, are you actually using group scheduling?
> 
> Not, yet - I wanted to start using it...
> 
> > could you try to reproduce without that?
> > 
> > CONFIG_GROUP_SCHED=n
> 
> I'll try. 
> 
> Do you think it's source of a problem?

On these old (.27 is ancient) kernels it might well be. In any case its
a good test to do, the cgroup code is significant in size and knowing on
which side of that border to look helps.

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4298 bytes --]

On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > could you try to reproduce without that?
> > > 
> > > CONFIG_GROUP_SCHED=n
> > I'll try. 

It seems that problem still exists - system has crashed too. From 
netconsole:
[20621.263297] BUG: unable to handle kernel paging request at ffffffd8
[20621.263396] IP: [<c081cc50>] hrtick_start_fair+0x0/0x30
[20621.263480] Oops: 0000 [#1] SMP 
[20621.263525] Modules linked in: configs softdog netconsole configfs sch_sfq xfs raid10 eepro100 uhci_hcd ppdev parport_pc parport floppy i2c_piix4 ehci_hcd piix e100 psmouse thermal button container e1000 mii i2c_core ide_core pcspkr serio_raw evdev processor sg usbcore ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[20621.264015] 
[20621.264042] Pid: 25390, comm: nnrpd Not tainted (2.6.27.31-1 #1)
[20621.264077] EIP: 0060:[<c081cc50>] EFLAGS: 00210046 CPU: 0
[20621.264116] EIP is at hrtick_start_fair+0x0/0x30
[20621.264148] EAX: c1d8a1e0 EBX: c1d8a1e0 ECX: 00000003 EDX: ffffffd4
[20621.264183] ESI: 00000000 EDI: ffffffd4 EBP: c1d8a1e0 ESP: f33d3b38
[20621.264218]  DS: 0068 ES: 007b FS: 00d8 GS: 0033 SS: 0068
[20621.264253] Process nnrpd (pid: 25390, ti=f33d2000 task=ee635640 task.ti=f33d2000)
[20621.264290] Stack: c081cd5b 00000000 00000000 c1d8a660 c0acaf00 00000100 00000000 c1d8a1e0 
[20621.264403]        c0a9beb3 f33d3bb8 00000000 00000000 00000000 00000000 ee635640 c1d8a1e0 
[20621.264515]        c1d8a668 00000000 00000000 00000000 00000000 00000000 000021e0 ee6357c8 
[20621.264619] Call Trace:
[20621.264619]  [<c081cd5b>] pick_next_task_fair+0xdb/0x100
[20621.264619]  [<c0a9beb3>] schedule+0x5b3/0xc00
[20621.264619]  [<c0830827>] lock_timer_base+0x27/0x50
[20621.264619]  [<c08309b9>] __mod_timer+0xa9/0xf0
[20621.264619]  [<c09480cd>] gr_log_resource+0x2d/0xf0
[20621.264619]  [<c0a9c84d>] schedule_timeout+0x8d/0xf0
[20621.264619]  [<c0830430>] process_timeout+0x0/0x10
[20621.264619]  [<c0a9c848>] schedule_timeout+0x88/0xf0
[20621.264619]  [<c08b231d>] do_sys_poll+0x30d/0x390
[20621.264619]  [<f89dedd0>] init_module+0xcdd0/0xcfe0 [sch_sfq]
[20621.264619]  [<c08b3300>] __pollwait+0x0/0x100
[20621.264619]  [<c08209c0>] default_wake_function+0x0/0x10
[20621.264619]  [<c0a110f2>] ip_finish_output+0x122/0x2a0
[20621.264619]  [<c0a0eafa>] ip_cork_release+0x2a/0x40
[20621.264619]  [<c0a105f1>] ip_push_pending_frames+0x2a1/0x360
[20621.264619]  [<c0a2e6bf>] udp_push_pending_frames+0x14f/0x3a0
[20621.264619]  [<c0a9e2d8>] _spin_lock_bh+0x8/0x20
[20621.264619]  [<c09de302>] release_sock+0x12/0xa0
[20621.264619]  [<c0a2fe9f>] udp_sendmsg+0x1af/0x680
[20621.264619]  [<c08a094f>] mem_cgroup_charge_common+0x1f/0x2c0
[20621.264619]  [<c087723d>] find_get_page+0x1d/0x90
[20621.264619]  [<c0a36809>] inet_sendmsg+0x29/0x60
[20621.264619]  [<c09db8d1>] sock_sendmsg+0x111/0x1b0
[20621.264619]  [<c083b080>] autoremove_wake_function+0x0/0x40
[20621.264619]  [<c0887d72>] handle_mm_fault+0x1c2/0x9b0
[20621.264619]  [<c0a08f47>] ip_route_output_flow+0x67/0x2b0
[20621.264619]  [<c08476d8>] ip_v4_find_src+0x88/0x330
[20621.264619]  [<c0a2becb>] ip4_datagram_connect+0x22b/0x2b0
[20621.264619]  [<c0aa060d>] do_page_fault+0x44d/0x740
[20621.264619]  [<c0a36866>] inet_dgram_connect+0x26/0x80
[20621.264619]  [<c09dc85e>] sys_connect+0xee/0x100
[20621.264619]  [<c08d36ef>] inotify_d_instantiate+0xf/0x40
[20621.264619]  [<c08b7fa6>] d_instantiate+0x46/0x70
[20621.264619]  [<c09dc767>] sys_send+0x37/0x40
[20621.264619]  [<c09dd427>] sys_socketcall+0x1c7/0x2b0
[20621.264619]  [<c08b2659>] sys_poll+0x39/0x90
[20621.264619]  [<c0803d54>] sysenter_do_call+0x12/0x21
[20621.264619]  =======================
[20621.264619] Code: 5e 5f 5d c3 8d b4 26 00 00 00 00 8d 73 3c 89 f0 e8 56 ef ff ff 8b 7f 48 85 ff 74 dc 89 ea 89 f0 e8 a6 fe ff ff eb d1 8d 74 26 00 <8b> 52 04 b9 e0 21 00 00 8b 52 10 8b 14 95 c0 f6 c0 c0 8d

Any ideas?

Last kernel I was using is 2.6.27.13 - works fine. None between 13 and 31 
tested...

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Tue, 2009-08-25 at 23:23 +0200, Pawel Golaszewski wrote:
> On Tue, 25 Aug 2009, Peter Zijlstra wrote:
> > > > could you try to reproduce without that?
> > > > 
> > > > CONFIG_GROUP_SCHED=n
> > > I'll try. 
> 
> It seems that problem still exists - system has crashed too. From 
> netconsole:

> Any ideas?
> 
> Last kernel I was using is 2.6.27.13 - works fine. None between 13 and 31 
> tested...

# git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation

Nothing changed anywhere near the code that is falling apart..

Also:

[20621.263297] BUG: unable to handle kernel paging request at ffffffd8
[20621.263396] IP: [<c081cc50>] hrtick_start_fair+0x0/0x30


#ifndef CONFIG_FAIR_GROUP_SCHED

static inline struct cfs_rq *cfs_rq_of(struct sched_entity *se)
{
        struct task_struct *p = task_of(se);
        struct rq *rq = task_rq(p);

        return &rq->cfs;
}

#endif

SCHED_FEAT(HRTICK, 0)

static void hrtick_start_fair(struct rq *rq, struct task_struct *p)
{
        struct sched_entity *se = &p->se;
        struct cfs_rq *cfs_rq = cfs_rq_of(se);

        WARN_ON(task_rq(p) != rq);

        if (hrtick_enabled(rq) && cfs_rq->nr_running > 1) {

		/* won't get here since HRTICK is disabled by default */

        }
}

Suggests something went terribly wrong and corrupted a fundamental data
structure.

Maybe turning on each and every debug feature related to memory
debugging might help.

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1356 bytes --]

On Wed, 26 Aug 2009, Peter Zijlstra wrote:
> > > > > could you try to reproduce without that?
> > > > > 
> > > > > CONFIG_GROUP_SCHED=n
> > > > I'll try. 
> > It seems that problem still exists - system has crashed too. From 
> > netconsole: Any ideas? Last kernel I was using is 2.6.27.13 - works 
> > fine. None between 13 and 31 tested...
> # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> 
> Nothing changed anywhere near the code that is falling apart..

I have 2 machines with the same hardware and similar software.
On one .13 is stable - I will test it on the second one. Should be too.

> Also:
> 
> [20621.263297] BUG: unable to handle kernel paging request at ffffffd8
> [20621.263396] IP: [<c081cc50>] hrtick_start_fair+0x0/0x30
[...]
> Suggests something went terribly wrong and corrupted a fundamental data 
> structure. Maybe turning on each and every debug feature related to 
> memory debugging might help.

ok, I'm a monkey - what can I do? :)

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Paweł Gołaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1535 bytes --]

On Wed, 26 Aug 2009, Pawel Golaszewski wrote:
> > > > > > could you try to reproduce without that?
> > > > > > 
> > > > > > CONFIG_GROUP_SCHED=n
> > > > > I'll try. 
> > > It seems that problem still exists - system has crashed too. From 
> > > netconsole: Any ideas? Last kernel I was using is 2.6.27.13 - works 
> > > fine. None between 13 and 31 tested...
> > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> > 
> > Nothing changed anywhere near the code that is falling apart..
> 
> I have 2 machines with the same hardware and similar software. On one 
> .13 is stable - I will test it on the second one. Should be too.

I've checked it - 2.6.27.13 is stable for me.

Conclusion: there is something wrong between 2.6.27.13 and 2.6.27.31

What can I do about that? I'm not kernel-hacker...

> > Also:
> > 
> > [20621.263297] BUG: unable to handle kernel paging request at ffffffd8
> > [20621.263396] IP: [<c081cc50>] hrtick_start_fair+0x0/0x30
> [...]
> > Suggests something went terribly wrong and corrupted a fundamental 
> > data structure. Maybe turning on each and every debug feature related 
> > to memory debugging might help.

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Thu, 2009-08-27 at 09:58 +0200, Paweł Gołaszewski wrote:
> On Wed, 26 Aug 2009, Pawel Golaszewski wrote:
> > > > > > > could you try to reproduce without that?
> > > > > > > 
> > > > > > > CONFIG_GROUP_SCHED=n
> > > > > > I'll try. 
> > > > It seems that problem still exists - system has crashed too. From 
> > > > netconsole: Any ideas? Last kernel I was using is 2.6.27.13 - works 
> > > > fine. None between 13 and 31 tested...
> > > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> > > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> > > 
> > > Nothing changed anywhere near the code that is falling apart..
> > 
> > I have 2 machines with the same hardware and similar software. On one 
> > .13 is stable - I will test it on the second one. Should be too.
> 
> I've checked it - 2.6.27.13 is stable for me.
> 
> Conclusion: there is something wrong between 2.6.27.13 and 2.6.27.31
> 
> What can I do about that? I'm not kernel-hacker...

Unless any of the memory debugging options yield a clue the best you can
do is a bisection I'm afraid.

# git log --format=oneline v2.6.27.13..v2.6.27.31 | wc -l
630

Which should be something like 10 kernel builds to find a patch, and
then one more with just that one reverted.

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1583 bytes --]

On Thu, 27 Aug 2009, Peter Zijlstra wrote:
> > > > > > > > could you try to reproduce without that?
> > > > > > > > 
> > > > > > > > CONFIG_GROUP_SCHED=n
> > > > > > > I'll try.
> > > > > It seems that problem still exists - system has crashed too. 
> > > > > From netconsole: Any ideas? Last kernel I was using is 2.6.27.13 
> > > > > - works fine. None between 13 and 31 tested...
> > > > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> > > > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> > > > 
> > > > Nothing changed anywhere near the code that is falling apart..
> > > I have 2 machines with the same hardware and similar software. On 
> > > one .13 is stable - I will test it on the second one. Should be too.
> > I've checked it - 2.6.27.13 is stable for me. Conclusion: there is 
> > something wrong between 2.6.27.13 and 2.6.27.31 What can I do about 
> > that? I'm not kernel-hacker...
> Unless any of the memory debugging options yield a clue

Which options?

> the best you can do is a bisection I'm afraid.
> 
> # git log --format=oneline v2.6.27.13..v2.6.27.31 | wc -l
> 630
> 
> Which should be something like 10 kernel builds to find a patch, and 
> then one more with just that one reverted.

I will try it too...

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Peter Zijlstra]

On Thu, 2009-08-27 at 11:24 +0200, Pawel Golaszewski wrote:
> On Thu, 27 Aug 2009, Peter Zijlstra wrote:
> > > > > > > > > could you try to reproduce without that?
> > > > > > > > > 
> > > > > > > > > CONFIG_GROUP_SCHED=n
> > > > > > > > I'll try.
> > > > > > It seems that problem still exists - system has crashed too. 
> > > > > > From netconsole: Any ideas? Last kernel I was using is 2.6.27.13 
> > > > > > - works fine. None between 13 and 31 tested...
> > > > > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> > > > > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> > > > > 
> > > > > Nothing changed anywhere near the code that is falling apart..
> > > > I have 2 machines with the same hardware and similar software. On 
> > > > one .13 is stable - I will test it on the second one. Should be too.
> > > I've checked it - 2.6.27.13 is stable for me. Conclusion: there is 
> > > something wrong between 2.6.27.13 and 2.6.27.31 What can I do about 
> > > that? I'm not kernel-hacker...
> > Unless any of the memory debugging options yield a clue
> 
> Which options?

I'm not sure what was available in .27

looking at your config, the things you can try are:

CONFIG_SLUB_DEBUG_ON=y
CONFIG_DEBUG_PAGEALLOC=y

I'm not seeing kmemcheck.

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 9531 bytes --]

On Thu, 27 Aug 2009, Peter Zijlstra wrote:
> > > > > > > > could you try to reproduce without that?
> > > > > > > > 
> > > > > > > > CONFIG_GROUP_SCHED=n
> > > > > > > I'll try. 
> > > > > It seems that problem still exists - system has crashed too. From 
> > > > > netconsole: Any ideas? Last kernel I was using is 2.6.27.13 - works 
> > > > > fine. None between 13 and 31 tested...
> > > > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> > > > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> > > > 
> > > > Nothing changed anywhere near the code that is falling apart..
> > > I have 2 machines with the same hardware and similar software. On 
> > > one .13 is stable - I will test it on the second one. Should be too.
> > I've checked it - 2.6.27.13 is stable for me.
> > 
> > Conclusion: there is something wrong between 2.6.27.13 and 2.6.27.31 
> > What can I do about that? I'm not kernel-hacker...
> Unless any of the memory debugging options yield a clue the best you can 
> do is a bisection I'm afraid.
> 
> # git log --format=oneline v2.6.27.13..v2.6.27.31 | wc -l
> 630

Checked 2.6.27.25 - crashed. Problem is somewhere between 2.6.27.13 and 
2.6.27.25. I will check one or two more kernels - sorry, I can't play more 
on that machine...
>From console - will it help?:

[ 4243.524886] BUG: unable to handle kernel <1>BUG: unable to handle kernel paging request at ffffffd8
[ 4243.524935] IP: [<c081cc50>] hrtick_start_fair+0x0/0x30
[ 4243.524991] *pde = 00008067 *pte = 00000000 
[ 4243.525006] Oops: 0000 [#1] SMP 
[ 4243.525017] Modules linked in: softdog netconsole configfs sch_sfq xfs raid10 ppdev uhci_hcd eepro100 parport_pc ehci_hcd parport piix psmouse e100 i2c_piix4 serio_raw evdev thermal usbcore e1000 mii ide_core container i2c_core sg pcspkr processor button ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[ 4243.525087] 
[ 4243.525097] Pid: 1783, comm: xfsbufd Not tainted (2.6.27.25-1 #1)
[ 4243.525105] EIP: 0060:[<c081cc50>] EFLAGS: 00010046 CPU: 0
[ 4243.525118] EIP is at hrtick_start_fair+0x0/0x30
[ 4243.525125] EAX: c1d8be00 EBX: c1d8be00 ECX: 00000001 EDX: ffffffd4
[ 4243.525133] ESI: 00000000 EDI: ffffffd4 EBP: c1d8be00 ESP: f72afec4
[ 4243.525140]  DS: 0068 ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 4243.525148] Process xfsbufd (pid: 1783, ti=f72ae000 task=f78af980 task.ti=f72ae000)
[ 4243.525157] Stack: c081cd5b 00000000 c1d8be00 c1d8be00 c0ac9f00 f78af980 00000000 c1d8be00 
[ 4243.525171]        c0a9b593 00000000 00000082 f88b5044 00000004 f72aff14 f78af980 c1d8be00 
[ 4243.525184]        c1d8c288 000003db 00000000 00000000 f72514f0 00200286 00003e00 f78afb08 
[ 4243.525198] Call Trace:
[ 4243.525205]  [<c081cd5b>] pick_next_task_fair+0xdb/0x100
[ 4243.525220]  [<c0a9b593>] schedule+0x5b3/0xc00
[ 4243.525255]  [<f88b5044>] ahc_print_path+0xba4/0x1090 [aic7xxx]
[ 4243.525313]  [<c0830837>] lock_timer_base+0x27/0x50
[ 4243.525332]  [<c08309c9>] __mod_timer+0xa9/0xf0
[ 4243.525341]  [<c08308a1>] try_to_del_timer_sync+0x41/0x50
[ 4243.525351]  [<c0a9bf2d>] schedule_timeout+0x8d/0xf0
[ 4243.525363]  [<c0830440>] process_timeout+0x0/0x10
[ 4243.525372]  [<c0a9bf28>] schedule_timeout+0x88/0xf0
[ 4243.525382]  [<f8b0f3df>] xfs_free_buftarg+0x9f/0x160 [xfs]
[ 4243.525523]  [<f8b0f380>] xfs_free_buftarg+0x40/0x160 [xfs]
[ 4243.525554]  [<c083ade9>] kthread+0x39/0x70
[ 4243.525575]  [<c083adb0>] kthread+0x0/0x70
[ 4243.525583]  [<c0805007>] kernel_thread_helper+0x7/0x10
[ 4243.525603]  =======================
[ 4243.525607] Code: 5e 5f 5d c3 8d b4 26 00 00 00 00 8d 73 3c 89 f0 e8 56 ef ff ff 8b 7f 48 85 ff 74 dc 89 ea 89 f0 e8 a6 fe ff ff eb d1 8d 74 26 00 <8b> 52 04 b9 00 3e 00 00 8b 52 10 8b 14 95 c0 f6 c0 c0 8d 94 11 
[ 4243.525670] EIP: [<c081cc50>] hrtick_start_fair+0x0/0x30 SS:ESP 0068:f72afec4
[ 4243.526080] Kernel panic - not syncing: Fatal exception
[ 4243.526120] ------------[ cut here ]------------
[ 4243.526127] WARNING: at kernel/smp.c:332 smp_call_function_mask+0x16d/0x1c0()
[ 4243.526133] Modules linked in: softdog netconsole configfs sch_sfq xfs raid10 ppdev uhci_hcd eepro100 parport_pc ehci_hcd parport piix psmouse e100 i2c_piix4 serio_raw evdev thermal usbcore e1000 mii ide_core container i2c_core sg pcspkr processor button ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[ 4243.526192] Pid: 1783, comm: xfsbufd Tainted: G      D   2.6.27.25-1 #1
[ 4243.526199]  [<c082629f>] warn_on_slowpath+0x5f/0xa0
[ 4243.526223]  [<c0a9ae54>] printk+0x17/0x1b
[ 4243.526233]  [<c08408df>] sched_clock_cpu+0x11f/0x180
[ 4243.526252]  [<c083f925>] down_trylock+0x25/0x40
[ 4243.526261]  [<c083f925>] down_trylock+0x25/0x40
[ 4243.526269]  [<c08268fa>] try_acquire_console_sem+0xa/0x30
[ 4243.526281]  [<c085434d>] smp_call_function_mask+0x16d/0x1c0
[ 4243.526291]  [<c085f85c>] crash_kexec+0x6c/0xd0
[ 4243.526316]  [<c0850020>] tick_nohz_stop_sched_tick+0x300/0x390
[ 4243.526326]  [<c0830837>] lock_timer_base+0x27/0x50
[ 4243.526335]  [<c0967eee>] vga_set_palette+0xbe/0x110
[ 4243.526362]  [<c0967eee>] vga_set_palette+0xbe/0x110
[ 4243.526372]  [<c08543b4>] smp_call_function+0x14/0x20
[ 4243.526381]  [<c081249e>] native_smp_send_stop+0x1e/0x30
[ 4243.526406]  [<c0a9adab>] panic+0x51/0xe3
[ 4243.526415]  [<c0a9df34>] oops_end+0xa4/0xb0
[ 4243.526429]  [<c0a9fc08>] do_page_fault+0x368/0x730
[ 4243.526443]  [<c089bbc4>] slab_pad_check+0x44/0x140
[ 4243.526474]  [<c089c0df>] check_object+0xdf/0x210
[ 4243.526485]  [<c087a220>] mempool_alloc+0x40/0x100
[ 4243.526508]  [<c089dbfa>] __slab_alloc+0x4aa/0x590
[ 4243.526520]  [<c089c0df>] check_object+0xdf/0x210
[ 4243.526529]  [<c087a220>] mempool_alloc+0x40/0x100
[ 4243.526538]  [<c087a220>] mempool_alloc+0x40/0x100
[ 4243.526547]  [<c087a220>] mempool_alloc+0x40/0x100
[ 4243.526557]  [<f8858020>] scsi_free_command+0x80/0xb0 [scsi_mod]
[ 4243.526615]  [<c0951cbf>] __sg_alloc_table+0x5f/0xf0
[ 4243.526632]  [<c0a9f8a0>] do_page_fault+0x0/0x730
[ 4243.526641]  [<c0a9dbc5>] error_code+0x75/0x80
[ 4243.526650]  [<c081cc50>] hrtick_start_fair+0x0/0x30
[ 4243.526660]  [<c081cd5b>] pick_next_task_fair+0xdb/0x100
[ 4243.526670]  [<c0a9b593>] schedule+0x5b3/0xc00
[ 4243.526681]  [<f88b5044>] ahc_print_path+0xba4/0x1090 [aic7xxx]
[ 4243.526701]  [<c0830837>] lock_timer_base+0x27/0x50
[ 4243.526710]  [<c08309c9>] __mod_timer+0xa9/0xf0
[ 4243.526718]  [<c08308a1>] try_to_del_timer_sync+0x41/0x50
[ 4243.526727]  [<c0a9bf2d>] schedule_timeout+0x8d/0xf0
[ 4243.526737]  [<c0830440>] process_timeout+0x0/0x10
[ 4243.526745]  [<c0a9bf28>] schedule_timeout+0x88/0xf0
[ 4243.526755]  [<f8b0f3df>] xfs_free_buftarg+0x9f/0x160 [xfs]
[ 4243.526786]  [<f8b0f380>] xfs_free_buftarg+0x40/0x160 [xfs]
[ 4243.526816]  [<c083ade9>] kthread+0x39/0x70
[ 4243.526825]  [<c083adb0>] kthread+0x0/0x70
[ 4243.526833]  [<c0805007>] kernel_thread_helper+0x7/0x10
[ 4243.526842]  =======================
[ 4243.526847] ---[ end trace bce437568718debf ]---
[ 4243.526871] Rebooting in 10 seconds..NULL pointer dereference at 00000000
[ 4243.527499] IP: [<c094c633>] rb_insert_color+0x53/0x100
[ 4243.527499] *pde = 00000000 
[ 4243.527499] Oops: 0002 [#2] SMP 
[ 4243.527499] Modules linked in: softdog netconsole configfs sch_sfq xfs raid10 ppdev uhci_hcd eepro100 parport_pc ehci_hcd parport piix psmouse e100 i2c_piix4 serio_raw evdev thermal usbcore e1000 mii ide_core container i2c_core sg pcspkr processor button ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[ 4243.527499] 
[ 4243.527499] Pid: 207, comm: kswapd0 Tainted: G      D W (2.6.27.25-1 #1)
[ 4243.527499] EIP: 0060:[<c094c633>] EFLAGS: 00010002 CPU: 3
[ 4243.527499] EIP is at rb_insert_color+0x53/0x100
[ 4243.527499] EAX: 00000000 EBX: c1d8be00 ECX: f78af9b4 EDX: c1d8be60
[ 4243.527499] ESI: f71c9b00 EDI: f78af9b4 EBP: f71c9b34 ESP: f7a47e6c
[ 4243.527499]  DS: 0068 ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 4243.527499] Process kswapd0 (pid: 207, ti=f7a46000 task=f7922400 task.ti=f7a46000)
[ 4243.527499] Stack: c1d8be3c c1d8be60 c1d8be00 f71c9b00 c1d8be3c f71c9b2c c081da54 00000000 
[ 4243.527499]        c1da0e00 c0ac9f00 f71c9b00 05ca51ee 000003dc c081ba46 c1da0e00 f71cb214 
[ 4243.527499]        f71c9b00 0000012c c0a9b560 f7a47f14 00000086 00000079 c1d8be00 00000001 
[ 4243.527499] Call Trace:
[ 4243.527499]  [<c081da54>] enqueue_task_fair+0x94/0xe0
[ 4243.527499]  [<c081ba46>] enqueue_task+0x26/0x80
[ 4243.527499]  [<c0a9b560>] schedule+0x580/0xc00
[ 4243.527499]  [<c0882c2f>] kswapd+0x4df/0x520
[ 4243.527499]  [<c0881270>] isolate_pages_global+0x0/0x60
[ 4243.527499]  [<c083b090>] autoremove_wake_function+0x0/0x40
[ 4243.527499]  [<c0882750>] kswapd+0x0/0x520
[ 4243.527499]  [<c083ade9>] kthread+0x39/0x70
[ 4243.527499]  [<c083adb0>] kthread+0x0/0x70
[ 4243.527499]  [<c0805007>] kernel_thread_helper+0x7/0x10
[ 4243.527499]  =======================
[ 4243.527499] Code: 74 5e 85 db 74 32 8b 13 f6 c2 01 75 2b 83 ca 01 89 13 89 f3 83 0f 01 89 dd 8b 3e 83 e7 fe 89 3e 83 e7 fc 75 cb 8b 54 24 04 8b 02 <83> 08 01 83 c4 08 5b 5e 5f 5d c3


-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Paweł Gołaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3540 bytes --]

On Thu, 27 Aug 2009, Pawel Golaszewski wrote:
> Checked 2.6.27.25 - crashed. Problem is somewhere between 2.6.27.13 and 
> 2.6.27.25.

2.6.27.17 crashed too, but a bit differently:
[...]
[  773.097432] BUG: soft lockup - CPU#2 stuck for 61s! [md2_raid10:1683]
[  773.097432] Kernel panic - not syncing: softlockup: hung tasks
[  773.097432] ------------[ cut here ]------------
[  773.097432] WARNING: at kernel/mutex.c:351 mutex_trylock+0xa5/0xd0()
[  773.097432] Modules linked in: softdog netconsole configfs sch_sfq xfs raid10 ppdev uhci_hcd eepro100 ehci_hcd parport_pc parport i2c_piix4 piix e100 thermal usbcore ide_core container e1000 i2c_core psmouse mii sg pcspkr button processor serio_raw evdev ext3 jbd mbcache scsi_wait_scan sd_mod crc_t10dif ata_piix libata dock aic7xxx sym53c8xx scsi_transport_spi scsi_mod raid1 md_mod
[  773.099637] Pid: 1683, comm: md2_raid10 Not tainted 2.6.27.17-1 #1
[  773.099637]  [<c0826dff>] warn_on_slowpath+0x5f/0xa0
[  773.099637]  [<f8a231c1>] __this_module+0x21fe1/0x227f0 [netconsole]
[  773.099637]  [<f8a23130>] __this_module+0x21f50/0x227f0 [netconsole]
[  773.099637]  [<c0827126>] __call_console_drivers+0x56/0x70
[  773.099637]  [<c084045b>] up+0xb/0x40
[  773.099637]  [<c0827673>] release_console_sem+0x1b3/0x1f0
[  773.099637]  [<c0a9bb95>] mutex_trylock+0xa5/0xd0
[  773.099637]  [<c086032f>] crash_kexec+0x1f/0xd0
[  773.099637]  [<c0831347>] lock_timer_base+0x27/0x50
[  773.099637]  [<c0967e8e>] vga_set_palette+0xbe/0x110
[  773.099637]  [<c0967e8e>] vga_set_palette+0xbe/0x110
[  773.099637]  [<c0968333>] vgacon_set_cursor_size+0x93/0x130
[  773.099637]  [<c0a9a957>] panic+0x4b/0xe3
[  773.099637]  [<c0871031>] softlockup_tick+0x191/0x1a0
[  773.099637]  [<c08310ec>] update_process_times+0x2c/0x60
[  773.099637]  [<c08505eb>] tick_sched_timer+0x6b/0xe0
[  773.099637]  [<c083f037>] __run_hrtimer+0x97/0xb0
[  773.099637]  [<c0850580>] tick_sched_timer+0x0/0xe0
[  773.099637]  [<c083fc7f>] hrtimer_interrupt+0x17f/0x1f0
[  773.099637]  [<c08134d8>] smp_apic_timer_interrupt+0x58/0x90
[  773.099637]  [<c0804e48>] apic_timer_interrupt+0x28/0x30
[  773.099637]  [<f8a29fed>] cleanup_module+0x2bdd/0x478b [raid10]
[  773.099637]  [<c0a9b85d>] schedule_timeout+0xbd/0xf0
[  773.099637]  [<c0804d53>] common_interrupt+0x23/0x30
[  773.099637]  [<c083bd11>] finish_wait+0x31/0x80
[  773.099637]  [<f882d479>] md_register_thread+0xc9/0x180 [md_mod]
[  773.099637]  [<c083bba0>] autoremove_wake_function+0x0/0x40
[  773.099637]  [<f882d450>] md_register_thread+0xa0/0x180 [md_mod]
[  773.099637]  [<c083b8f9>] kthread+0x39/0x70
[  773.099637]  [<c083b8c0>] kthread+0x0/0x70
[  773.099637]  [<c0805007>] kernel_thread_helper+0x7/0x10
[  773.099637]  =======================
[  773.099637] ---[ end trace 5cfcde0c2c70ce60 ]---
[  773.099637] ------------[ cut here ]------------
[  773.099637] WARNING: at kernel/smp.c:332 smp_call_function_mask+0x16d/0x1c0()
[  773.099637] Modules linked in: softdog netconsole configfs sch_sfq xfs raid10 ppdev uhci_hcd eepro100 ehci_hcd parport_pc parport i2c_piix4 piix e100 thermal usbcore ide_core container e1000 i2c_core psmouse mii sg pcspkr button processor serio_raw evdev ext3 jbd mbcache
[...]

Problem is somewhere 2.6.27.13-2.6.27.17. 

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Pawel Golaszewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1168 bytes --]

On Thu, 27 Aug 2009, Peter Zijlstra wrote:
> > > > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
> > > > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
> > > > 
> > > > Nothing changed anywhere near the code that is falling apart..
> > > I have 2 machines with the same hardware and similar software. On 
> > > one .13 is stable - I will test it on the second one. Should be too.
> > I've checked it - 2.6.27.13 is stable for me. Conclusion: there is 
> > something wrong between 2.6.27.13 and 2.6.27.31 What can I do about 
> > that? I'm not kernel-hacker...
> Unless any of the memory debugging options yield a clue the best you can 
> do is a bisection I'm afraid.
> 
> # git log --format=oneline v2.6.27.13..v2.6.27.31 | wc -l
> 630

Ok, I checked:

2.6.27.15 works fine
2.6.27.17 crashes (log was sent).

What now?

-- 
pozdr.  Paweł Gołaszewski          jid:blues<at>jabber<dot>gda<dot>pl
--------------------------------------------------------------------------
If you think of MS-DOS as mono, and Windows as stereo, then Linux is Dolby
Pro-Logic Surround Sound with Bass Boost and all the music is free.

Re: PROBLEM: oops

[Eric Paris]

On Fri, Aug 28, 2009 at 5:46 PM, Pawel Golaszewski<blues@gda.pl> wrote:
> On Thu, 27 Aug 2009, Peter Zijlstra wrote:
>> > > > # git log --format=oneline v2.6.27.13..v2.6.27.31 kernel/sched*
>> > > > 2b46f3769896dc04e1e49144d282e4655677105a wait: prevent exclusive waiter starvation
>> > > >
>> > > > Nothing changed anywhere near the code that is falling apart..
>> > > I have 2 machines with the same hardware and similar software. On
>> > > one .13 is stable - I will test it on the second one. Should be too.
>> > I've checked it - 2.6.27.13 is stable for me. Conclusion: there is
>> > something wrong between 2.6.27.13 and 2.6.27.31 What can I do about
>> > that? I'm not kernel-hacker...
>> Unless any of the memory debugging options yield a clue the best you can
>> do is a bisection I'm afraid.
>>
>> # git log --format=oneline v2.6.27.13..v2.6.27.31 | wc -l
>> 630
>
> Ok, I checked:
>
> 2.6.27.15 works fine
> 2.6.27.17 crashes (log was sent).
>
> What now?

Have you ever used git, and especially git-bisect?  That's what would
really help to run it down.  Try this operation.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/hpa/linux-2.6-allstable.git
stable-tree
cd stable-tree
git bisect start
git bisect bad v2.6.27.17
git bisect good v2.6.27.15

(a)
[git will do some work and will leave you with a message that says
something like "Bisecting: 675 revisions left to test after this"]
build the kernel tree.
test your build.

if it works go back to the stable-tree directory and enter:
git bisect good

if that kernel fails enter:
git bisect bad

goto (a)

Eventually there will be no more revisions left to bisect, and you
will have been left with the first bad kernel revision in
"refs/bisect/bad".

Let us know what this patch is that broke it for you.

Thanks!

Re: PROBLEM: oops

[Arkadiusz Miśkiewicz]

Peter Zijlstra wrote:

> I'm not sure what was available in .27
> 
> looking at your config, the things you can try are:
> 
> CONFIG_SLUB_DEBUG_ON=y
> CONFIG_DEBUG_PAGEALLOC=y
> 
> I'm not seeing kmemcheck.

I'm fighting the same problem as blues is. The problem, in my case, is 
visible on intel x86_64 and amd x86_64 machines (so different hardware than 
his i686).

I get oops on 2.6.27.31 and 2.6.27.30 (see below). 2.6.31 with debug options 
and kmemcheck didn't really work (see last log below).

I don't know how to trigger that oops. In my case it takes ~1 week for the 
oops to happen. These kernels aren't (unfortunately) vanilla. They have 
vserver patch applied. I'm also using grsecurity patch but I tested without 
it, too - oops did happen.

I'm unable to test these without vserver patch.

************************************************************************************
2.6.27.31 on intel machine:

[ 3952.746892] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000080
[ 3952.747315] IP: [<ffffffff80227e04>] pick_next_task_fair+0x64/0xf0
[ 3952.747315] PGD 2865eb067 PUD 2a3d60067 PMD 0 
[ 3952.747315] Oops: 0000 [1] SMP 
[ 3952.747315] CPU 5 
[ 3952.747315] Modules linked in: ipmi_watchdog sch_sfq dm_mod e1000e 
ipmi_devintf ipmi_si ipmi_msghandler xfs scsi_wait_scan sd_mod crc_t10dif 
aacraid scsi_mod
[ 3952.747315] Pid: 14053, comm: exim Not tainted 2.6.27.31-1 #1
[ 3952.747315] RIP: 0010:[<ffffffff80227e04>]  [<ffffffff80227e04>] 
pick_next_task_fair+0x64/0xf0
[ 3952.747315] RSP: 0018:ffff880286511a18  EFLAGS: 00210046
[ 3952.747315] RAX: 0000000000000000 RBX: ffff88002805fe70 RCX: 
ffff880314dda260
[ 3952.747315] RDX: ffff88035e4e3780 RSI: ffff880314dda260 RDI: 
ffff88002805fe00
[ 3952.747315] RBP: ffff880286511a58 R08: ffff8800280606e8 R09: 
0000000000000000
[ 3952.747315] R10: 0000000000000000 R11: 0000000000000001 R12: 
0000000000000000
[ 3952.747315] R13: ffff88035919a950 R14: 0000000000000000 R15: 
ffff88035919a950
[ 3952.747315] FS:  0000000000000000(0000) GS:ffff88035e4e3780(0063) 
knlGS:00000000f7217910
[ 3952.747315] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 3952.747315] CR2: 0000000000000080 CR3: 0000000286459000 CR4: 
00000000000006e0
[ 3952.747315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[ 3952.747315] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[ 3952.747315] Process exim (pid: 14053, threadinfo ffff880286510000, task 
ffff880314dda260)
[ 3952.747315] Stack:  ffff880286511a38 ffff88002805fe00 0000000000000005 
ffff88002805fe00
[ 3952.747315]  ffffffff80550b80 ffff88035919a950 ffff8800280606e8 
ffff88035919a950
[ 3952.747315]  ffff880286511b38 ffffffff8050d2ee 0000000000000000 
0000000000001000
[ 3952.747315] Call Trace:
[ 3952.747315]  [<ffffffff8050d2ee>] thread_return+0x29d/0x88f
[ 3952.747315]  [<ffffffff802f7dc1>] ? __block_prepare_write+0x2a1/0x540
[ 3952.747315]  [<ffffffff802f6442>] ? __set_page_dirty+0x92/0x150
[ 3952.747315]  [<ffffffff802f65ea>] ? __block_commit_write+0x6a/0xd0
[ 3952.747315]  [<ffffffff8050dcfa>] schedule_timeout+0x8a/0xd0
[ 3952.747315]  [<ffffffff802f67d9>] ? block_write_end+0x39/0x90
[ 3952.747315]  [<ffffffff8050f9e9>] ? _spin_lock_bh+0x9/0x20
[ 3952.747315]  [<ffffffff80436f53>] ? release_sock+0x13/0xb0
[ 3952.747315]  [<ffffffff804376d1>] ? sk_wait_data+0xe1/0xf0
[ 3952.747315]  [<ffffffff8024b220>] ? autoremove_wake_function+0x0/0x30
[ 3952.747315]  [<ffffffff80479cad>] ? tcp_recvmsg+0x65d/0xd90
[ 3952.747315]  [<ffffffffa00b6a60>] ? 
xfs_address_space_operations+0x0/0x1a0 [xfs]
[ 3952.747315]  [<ffffffff80436332>] ? sock_common_recvmsg+0x32/0x50
[ 3952.747315]  [<ffffffff80433582>] ? sock_aio_read+0x192/0x200
[ 3952.747315]  [<ffffffff802cb222>] ? do_sync_read+0xe2/0x120
[ 3952.747315]  [<ffffffff805120a9>] ? do_page_fault+0x339/0x960
[ 3952.747315]  [<ffffffff8024dfce>] ? enqueue_hrtimer+0x6e/0xf0
[ 3952.747315]  [<ffffffff8024b220>] ? autoremove_wake_function+0x0/0x30
[ 3952.747315]  [<ffffffff80238c38>] ? do_setitimer+0x2f8/0x3a0
[ 3952.747315]  [<ffffffff802cbf3a>] ? vfs_read+0x16a/0x180
[ 3952.747315]  [<ffffffff80238df5>] ? alarm_setitimer+0x35/0x70
[ 3952.747315]  [<ffffffff802cc03e>] ? sys_read+0x4e/0x90
[ 3952.747315]  [<ffffffff80222776>] ? sysenter_dispatch+0x7/0x27
[ 3952.747315] 
[ 3952.747315] 
[ 3952.747315] Code: 74 24 f0 4c 89 6b 28 45 8b 56 38 45 85 d2 75 68 48 8b 
43 70 48 8b 80 e0 07 00 00 49 89 46 40 4c 89 73 58 49 8b 46 48 49 89 46 58 
<49> 8b 9e 80 00 00 00 48 85 db 74 50 4c 8b 63 38 45 31 f6 4d 85 
[ 3952.747315] RIP  [<ffffffff80227e04>] pick_next_task_fair+0x64/0xf0
[ 3952.747315]  RSP <ffff880286511a18>
[ 3952.747315] CR2: 0000000000000080
[ 3952.747315] ---[ end trace 82f82983f40cc2e1 ]---

2.6.27.31 on amd machine:

[91371.385760] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000080
[91371.387743] IP: [<ffffffff80227e04>] pick_next_task_fair+0x64/0xf0
[91371.387743] PGD 217516067 PUD 217515067 PMD 0 
[91371.387743] Oops: 0000 [1] SMP 
[91371.387743] CPU 1 
[91371.387743] Modules linked in: sg xt_multiport xt_tcpudp 
nf_conntrack_ipv4 xt_state iptable_filter ip_tables ipt_ULOG x_tables 
ipmi_watchdog sch_sfq forcedeth ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
ff_memless nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si 
ipmi_msghandler ehci_hcd usbcore amd74xx ide_core xfs scsi_wait_scan sd_mod 
crc_t10dif aacraid scsi_mod
[91371.387743] Pid: 3710, comm: java Not tainted 2.6.27.31-1 #1
[91371.387743] RIP: 0010:[<ffffffff80227e04>]  [<ffffffff80227e04>] 
pick_next_task_fair+0x64/0xf0
[91371.387743] RSP: 0000:ffff88010b043b58  EFLAGS: 00010046
[91371.387743] RAX: 0000000000000000 RBX: ffff880028029e70 RCX: 
ffff88010b043c18
[91371.387743] RDX: 0000000000000011 RSI: ffff88002802a6d0 RDI: 
ffff880028029e00
[91371.387743] RBP: ffff88010b043b98 R08: ffff88002802a6e8 R09: 
0000000000000002
[91371.387743] R10: 0000000000000000 R11: 0000000000000000 R12: 
0000000000000000
[91371.387743] R13: ffff88011e9b0000 R14: 0000000000000000 R15: 
00000000ffffffff
[91371.387743] FS:  00007f626e5716f0(0000) GS:ffff88011fc02a80(0063) 
knlGS:00000000ade06bb0
[91371.387743] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[91371.387743] CR2: 0000000000000080 CR3: 00000002174f2000 CR4: 
00000000000006e0
[91371.387743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[91371.387743] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[91371.387743] Process java (pid: 3710, threadinfo ffff88010b042000, task 
ffff88011e64e720)
[91371.387743] Stack:  ffff88010b043b78 ffff880028029e00 0000000000000001 
ffff880028029e00
[91371.387743]  ffffffff80550b80 ffff88011e9b0000 ffff88002802a6e8 
00000000ffffffff
[91371.387743]  ffff88010b043c78 ffffffff8050d2ee 0000000000000096 
ffffffff80257eb8
[91371.387743] Call Trace:
[91371.387743]  [<ffffffff8050d2ee>] thread_return+0x29d/0x88f
[91371.387743]  [<ffffffff80257eb8>] ? vx_update_load+0x18/0x170
[91371.387743]  [<ffffffff80257eb8>] ? vx_update_load+0x18/0x170
[91371.387743]  [<ffffffff80228e05>] ? enqueue_entity+0xd5/0x120
[91371.387743]  [<ffffffff80263eb5>] futex_wait+0x435/0x4c0
[91371.387743]  [<ffffffff802275b3>] ? __wake_up+0x43/0x70
[91371.387743]  [<ffffffff802638ca>] ? get_futex_key+0x5a/0x180
[91371.387743]  [<ffffffff80394721>] ? __up_read+0x21/0xc0
[91371.387743]  [<ffffffff8022d3b0>] ? default_wake_function+0x0/0x10
[91371.387743]  [<ffffffff8026559f>] ? do_futex+0xff/0xaa0
[91371.387743]  [<ffffffff80227e70>] ? pick_next_task_fair+0xd0/0xf0
[91371.387743]  [<ffffffff8050d2ee>] ? thread_return+0x29d/0x88f
[91371.387743]  [<ffffffff8050d09e>] ? thread_return+0x4d/0x88f
[91371.387743]  [<ffffffff80266238>] ? compat_sys_futex+0xb8/0x160
[91371.387743]  [<ffffffff80222ba2>] ? ia32_sysret+0x0/0xa
[91371.387743] 
[91371.387743] 
[91371.387743] Code: 74 24 f0 4c 89 6b 28 45 8b 56 38 45 85 d2 75 68 48 8b 
43 70 48 8b 80 e0 07 00 00 49 89 46 40 4c 89 73 58 49 8b 46 48 49 89 46 58 
<49> 8b 9e 80 00 00 00 48 85 db 74 50 4c 8b 63 38 45 31 f6 4d 85 
[91371.387743] RIP  [<ffffffff80227e04>] pick_next_task_fair+0x64/0xf0
[91371.387743]  RSP <ffff88010b043b58>
[91371.387743] CR2: 0000000000000080
[91371.387743] ---[ end trace 70e3e5544a76f116 ]---

2.6.30 on amd machine:

[328109.962383] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000038
[328109.964692] IP: [<ffffffff802357eb>] pick_next_task_fair+0x9b/0x120
[328109.964692] PGD 216478067 PUD 2129cc067 PMD 0 
[328109.964692] Oops: 0000 [#1] SMP 
[328109.964692] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/irq
[328109.964692] CPU 4 
[328109.964692] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328109.964692] Pid: 16435, comm: php Not tainted 2.6.30.6_nogrsecurity-0.1 
#1 H8DMU
[328109.964692] RIP: 0010:[<ffffffff802357eb>]  [<ffffffff802357eb>] 
pick_next_task_fair+0x9b/0x120
[328109.964692] RSP: 0000:ffff8801ee9bd968  EFLAGS: 00210046
[328109.964692] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
ffff8801ee9bda08
[328109.964692] RDX: 000000000000008d RSI: 0000000000000001 RDI: 
0000000000000000
[328109.964692] RBP: ffffc20000061168 R08: ffffc20000061a08 R09: 
0000000000000000
[328109.964692] R10: 0000000000000000 R11: 0000000000000000 R12: 
0000000000000000
[328109.964692] R13: ffffc20000061100 R14: ffff880169b18a00 R15: 
ffffc20000061a18
[328109.964692] FS:  00007f4601b03790(0000) GS:ffffc20000050000(0063) 
knlGS:00000000f78716c0
[328109.964692] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[328109.964692] CR2: 0000000000000038 CR3: 000000012c2c5000 CR4: 
00000000000006e0
[328109.964692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328109.964692] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328109.964692] Process php (pid: 16435, threadinfo ffff8801ee9bc000, task 
ffff88021e97bcc0)
[328109.964692] Stack:
[328109.964692]  0000000000000000 ffffc20000061100 0000000000000000 
ffffffff80586aa0
[328109.964692]  ffffc20000061a18 ffffffff8056ed93 0000000000000004 
ffff8801655230f8
[328109.964692]  ffff88021f047000 ffffffff8025db8c ffff88021e97bcc0 
ffff8801ee9bda88
[328109.964692] Call Trace:
[328109.964692]  [<ffffffff8056ed93>] ? thread_return+0x26d/0x8ea
[328109.964692]  [<ffffffff8025db8c>] ? ktime_get_real+0xc/0x40
[328109.964692]  [<ffffffff8025cb1e>] ? enqueue_hrtimer+0x6e/0x80
[328109.964692]  [<ffffffff8025d454>] ? __hrtimer_start_range_ns+0x124/0x2b0
[328109.964692]  [<ffffffff8056f419>] ? schedule+0x9/0x20
[328109.964692]  [<ffffffff80570755>] ? schedule_hrtimeout_range+0xc5/0x160
[328109.964692]  [<ffffffff8025cbf0>] ? hrtimer_wakeup+0x0/0x30
[328109.964692]  [<ffffffff8057073b>] ? schedule_hrtimeout_range+0xab/0x160
[328109.964692]  [<ffffffff803021ec>] ? poll_schedule_timeout+0x2c/0x50
[328109.964692]  [<ffffffff8030263b>] ? do_sys_poll+0x37b/0x460
[328109.964692]  [<ffffffff804dbe0b>] ? tcp_write_xmit+0x1fb/0xa20
[328109.964692]  [<ffffffff803036c0>] ? __pollwait+0x0/0x120
[328109.964692]  [<ffffffff803037e0>] ? pollwake+0x0/0x50
[328109.964692]  [<ffffffff80482946>] ? sock_sendmsg+0x146/0x210
[328109.964692]  [<ffffffff80259d40>] ? autoremove_wake_function+0x0/0x30
[328109.964692]  [<ffffffff802af610>] ? find_lock_page+0x30/0x80
[328109.964692]  [<ffffffff802afd66>] ? filemap_fault+0x156/0x420
[328109.964692]  [<ffffffff802c5fea>] ? __do_fault+0x42a/0x510
[328109.964692]  [<ffffffff80483a8d>] ? sys_sendto+0x11d/0x180
[328109.964692]  [<ffffffff802c82e9>] ? handle_mm_fault+0x1b9/0xb00
[328109.964692]  [<ffffffff80484457>] ? sys_getsockname+0xd7/0xf0
[328109.964692]  [<ffffffff8026ea6e>] ? getnstimeofday+0x4e/0xe0
[328109.964692]  [<ffffffff8025d898>] ? ktime_get_ts+0x28/0x60
[328109.964692]  [<ffffffff80302194>] ? poll_select_set_timeout+0x74/0xa0
[328109.964692]  [<ffffffff80302921>] ? sys_poll+0x71/0x110
[328109.964692]  [<ffffffff80230272>] ? ia32_sysret+0x0/0xa
[328109.964692] Code: e4 ff ff 85 c0 7e 49 48 8b 7d 60 48 85 ff 74 38 48 89 
de e8 a8 e4 ff ff 85 c0 7e 44 48 8b 45 60 48 39 c3 74 3f 48 3b 5d 58 74 4d 
<8b> 73 38 85 f6 74 86 48 89 de 48 89 ef e8 93 f3 ff ff e9 76 ff 
[328109.964692] RIP  [<ffffffff802357eb>] pick_next_task_fair+0x9b/0x120
[328109.964692]  RSP <ffff8801ee9bd968>
[328109.964692] CR2: 0000000000000038
[328109.964692] ---[ end trace 22ebb73050974e75 ]---
[328192.410933] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328192.427185] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328192.533849] CPU 2:
[328192.540518] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328192.647185] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328192.673851] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328192.700518] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328192.717185] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328192.737185] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328192.760518] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328192.780518] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328192.803851] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328192.823851] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328192.847185] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328192.867185] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328192.887185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328192.910518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328192.930518] Call Trace:
[328192.937185]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328192.957185]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328192.977185]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328192.993851]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328193.013851]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328193.033851]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328193.050518]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328257.910925] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328257.927176] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328258.033843] CPU 2:
[328258.040509] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328258.147176] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328258.173843] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328258.200509] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328258.217176] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328258.237176] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328258.260509] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328258.280509] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328258.300509] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328258.323843] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328258.347176] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328258.367176] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328258.387176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328258.410509] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328258.430509] Call Trace:
[328258.437176]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328258.457176]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328258.477176]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328258.493843]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328258.513843]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328258.533843]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328258.550509]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328323.407585] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328323.423836] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328323.530503] CPU 2:
[328323.537170] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328323.643836] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328323.670503] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328323.697170] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328323.713836] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328323.733836] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328323.757170] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328323.777170] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328323.797170] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328323.820503] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328323.843836] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328323.863836] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328323.883836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328323.907170] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328323.927170] Call Trace:
[328323.933836]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328323.953836]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328323.973836]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328323.990503]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328324.010503]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328324.030503]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328324.047170]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328388.907578] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328388.923830] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328389.030496] CPU 2:
[328389.037163] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328389.143830] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328389.170496] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328389.197163] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328389.213830] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328389.233830] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328389.257163] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328389.277163] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328389.300496] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328389.320496] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328389.343830] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328389.363830] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328389.383830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328389.407163] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328389.427163] Call Trace:
[328389.433830]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328389.453830]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328389.473830]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328389.490496]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328389.510496]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328389.530496]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328389.547163]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328454.404238] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328454.420490] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328454.527157] CPU 2:
[328454.533823] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328454.640490] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328454.667157] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328454.693823] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328454.710490] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328454.730490] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328454.753823] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328454.773823] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328454.797157] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328454.817157] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328454.840490] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328454.860490] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328454.880490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328454.903823] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328454.923823] Call Trace:
[328454.930490]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328454.950490]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328454.970490]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328454.987156]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328455.007157]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328455.027157]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328455.043823]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328519.904232] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328519.920483] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328520.027150] CPU 2:
[328520.033817] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328520.140483] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328520.167150] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328520.193817] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328520.210483] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328520.230483] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328520.253817] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328520.273817] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328520.293817] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328520.317150] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328520.340483] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328520.360483] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328520.380483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328520.403817] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328520.423817] Call Trace:
[328520.430483]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328520.450483]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328520.470483]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328520.487150]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328520.507150]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328520.527150]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328520.543817]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328585.400892] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328585.417144] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328585.523810] CPU 2:
[328585.530477] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328585.637143] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328585.663810] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328585.690477] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328585.707143] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328585.727143] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328585.750477] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328585.770477] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328585.790477] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328585.813811] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328585.837144] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328585.857143] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328585.877143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328585.900477] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328585.920477] Call Trace:
[328585.927143]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328585.947143]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328585.967143]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328585.983810]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328586.003810]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328586.023810]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328586.040477]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328650.897552] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328650.913804] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328651.020470] CPU 2:
[328651.027137] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328651.133804] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328651.160470] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328651.187137] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328651.203803] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328651.223803] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328651.243803] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328651.267137] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328651.287137] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328651.310470] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328651.333804] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328651.353803] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328651.373803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328651.397137] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328651.417137] Call Trace:
[328651.423803]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328651.443803]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328651.463803]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328651.480470]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328651.500470]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328651.520470]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328651.537137]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328716.397545] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328716.413797] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328716.520464] CPU 2:
[328716.527130] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328716.633797] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328716.660464] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328716.687130] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328716.700464] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328716.723797] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328716.743797] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328716.767130] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328716.787130] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328716.810464] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328716.833797] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328716.853797] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328716.873797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328716.897130] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328716.917130] Call Trace:
[328716.923797]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328716.943797]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328716.963797]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328716.980463]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328717.000464]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328717.020464]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328717.037130]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328781.894206] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328781.910457] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328782.017124] CPU 2:
[328782.023791] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328782.130457] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328782.157124] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328782.183790] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328782.200457] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328782.220457] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328782.243790] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328782.263790] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328782.283790] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328782.307124] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328782.330457] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328782.350457] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328782.370457] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328782.393790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328782.413791] Call Trace:
[328782.420457]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328782.440457]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328782.460457]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328782.477124]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328782.497124]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328782.517124]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328782.533790]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30

************************************************************************************

2.6.30 on amd machine:

[328109.962383] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000038
[328109.964692] IP: [<ffffffff802357eb>] pick_next_task_fair+0x9b/0x120
[328109.964692] PGD 216478067 PUD 2129cc067 PMD 0 
[328109.964692] Oops: 0000 [#1] SMP 
[328109.964692] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/irq
[328109.964692] CPU 4 
[328109.964692] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328109.964692] Pid: 16435, comm: php Not tainted 2.6.30.6_nogrsecurity-0.1 
#1 H8DMU
[328109.964692] RIP: 0010:[<ffffffff802357eb>]  [<ffffffff802357eb>] 
pick_next_task_fair+0x9b/0x120
[328109.964692] RSP: 0000:ffff8801ee9bd968  EFLAGS: 00210046
[328109.964692] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
ffff8801ee9bda08
[328109.964692] RDX: 000000000000008d RSI: 0000000000000001 RDI: 
0000000000000000
[328109.964692] RBP: ffffc20000061168 R08: ffffc20000061a08 R09: 
0000000000000000
[328109.964692] R10: 0000000000000000 R11: 0000000000000000 R12: 
0000000000000000
[328109.964692] R13: ffffc20000061100 R14: ffff880169b18a00 R15: 
ffffc20000061a18
[328109.964692] FS:  00007f4601b03790(0000) GS:ffffc20000050000(0063) 
knlGS:00000000f78716c0
[328109.964692] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[328109.964692] CR2: 0000000000000038 CR3: 000000012c2c5000 CR4: 
00000000000006e0
[328109.964692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328109.964692] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328109.964692] Process php (pid: 16435, threadinfo ffff8801ee9bc000, task 
ffff88021e97bcc0)
[328109.964692] Stack:
[328109.964692]  0000000000000000 ffffc20000061100 0000000000000000 
ffffffff80586aa0
[328109.964692]  ffffc20000061a18 ffffffff8056ed93 0000000000000004 
ffff8801655230f8
[328109.964692]  ffff88021f047000 ffffffff8025db8c ffff88021e97bcc0 
ffff8801ee9bda88
[328109.964692] Call Trace:
[328109.964692]  [<ffffffff8056ed93>] ? thread_return+0x26d/0x8ea
[328109.964692]  [<ffffffff8025db8c>] ? ktime_get_real+0xc/0x40
[328109.964692]  [<ffffffff8025cb1e>] ? enqueue_hrtimer+0x6e/0x80
[328109.964692]  [<ffffffff8025d454>] ? __hrtimer_start_range_ns+0x124/0x2b0
[328109.964692]  [<ffffffff8056f419>] ? schedule+0x9/0x20
[328109.964692]  [<ffffffff80570755>] ? schedule_hrtimeout_range+0xc5/0x160
[328109.964692]  [<ffffffff8025cbf0>] ? hrtimer_wakeup+0x0/0x30
[328109.964692]  [<ffffffff8057073b>] ? schedule_hrtimeout_range+0xab/0x160
[328109.964692]  [<ffffffff803021ec>] ? poll_schedule_timeout+0x2c/0x50
[328109.964692]  [<ffffffff8030263b>] ? do_sys_poll+0x37b/0x460
[328109.964692]  [<ffffffff804dbe0b>] ? tcp_write_xmit+0x1fb/0xa20
[328109.964692]  [<ffffffff803036c0>] ? __pollwait+0x0/0x120
[328109.964692]  [<ffffffff803037e0>] ? pollwake+0x0/0x50
[328109.964692]  [<ffffffff80482946>] ? sock_sendmsg+0x146/0x210
[328109.964692]  [<ffffffff80259d40>] ? autoremove_wake_function+0x0/0x30
[328109.964692]  [<ffffffff802af610>] ? find_lock_page+0x30/0x80
[328109.964692]  [<ffffffff802afd66>] ? filemap_fault+0x156/0x420
[328109.964692]  [<ffffffff802c5fea>] ? __do_fault+0x42a/0x510
[328109.964692]  [<ffffffff80483a8d>] ? sys_sendto+0x11d/0x180
[328109.964692]  [<ffffffff802c82e9>] ? handle_mm_fault+0x1b9/0xb00
[328109.964692]  [<ffffffff80484457>] ? sys_getsockname+0xd7/0xf0
[328109.964692]  [<ffffffff8026ea6e>] ? getnstimeofday+0x4e/0xe0
[328109.964692]  [<ffffffff8025d898>] ? ktime_get_ts+0x28/0x60
[328109.964692]  [<ffffffff80302194>] ? poll_select_set_timeout+0x74/0xa0
[328109.964692]  [<ffffffff80302921>] ? sys_poll+0x71/0x110
[328109.964692]  [<ffffffff80230272>] ? ia32_sysret+0x0/0xa
[328109.964692] Code: e4 ff ff 85 c0 7e 49 48 8b 7d 60 48 85 ff 74 38 48 89 
de e8 a8 e4 ff ff 85 c0 7e 44 48 8b 45 60 48 39 c3 74 3f 48 3b 5d 58 74 4d 
<8b> 73 38 85 f6 74 86 48 89 de 48 89 ef e8 93 f3 ff ff e9 76 ff 
[328109.964692] RIP  [<ffffffff802357eb>] pick_next_task_fair+0x9b/0x120
[328109.964692]  RSP <ffff8801ee9bd968>
[328109.964692] CR2: 0000000000000038
[328109.964692] ---[ end trace 22ebb73050974e75 ]---
[328192.410933] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328192.427185] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328192.533849] CPU 2:
[328192.540518] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328192.647185] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328192.673851] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328192.700518] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328192.717185] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328192.737185] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328192.760518] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328192.780518] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328192.803851] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328192.823851] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328192.847185] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328192.867185] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328192.887185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328192.910518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328192.930518] Call Trace:
[328192.937185]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328192.957185]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328192.977185]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328192.993851]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328193.013851]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328193.033851]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328193.050518]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328257.910925] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328257.927176] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328258.033843] CPU 2:
[328258.040509] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328258.147176] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328258.173843] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328258.200509] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328258.217176] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328258.237176] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328258.260509] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328258.280509] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328258.300509] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328258.323843] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328258.347176] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328258.367176] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328258.387176] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328258.410509] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328258.430509] Call Trace:
[328258.437176]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328258.457176]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328258.477176]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328258.493843]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328258.513843]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328258.533843]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328258.550509]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328323.407585] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328323.423836] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328323.530503] CPU 2:
[328323.537170] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328323.643836] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328323.670503] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328323.697170] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328323.713836] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328323.733836] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328323.757170] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328323.777170] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328323.797170] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328323.820503] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328323.843836] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328323.863836] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328323.883836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328323.907170] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328323.927170] Call Trace:
[328323.933836]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328323.953836]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328323.973836]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328323.990503]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328324.010503]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328324.030503]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328324.047170]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328388.907578] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328388.923830] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328389.030496] CPU 2:
[328389.037163] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328389.143830] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328389.170496] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328389.197163] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328389.213830] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328389.233830] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328389.257163] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328389.277163] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328389.300496] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328389.320496] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328389.343830] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328389.363830] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328389.383830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328389.407163] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328389.427163] Call Trace:
[328389.433830]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328389.453830]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328389.473830]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328389.490496]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328389.510496]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328389.530496]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328389.547163]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328454.404238] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328454.420490] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328454.527157] CPU 2:
[328454.533823] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328454.640490] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328454.667157] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328454.693823] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328454.710490] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328454.730490] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328454.753823] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328454.773823] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328454.797157] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328454.817157] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328454.840490] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328454.860490] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328454.880490] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328454.903823] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328454.923823] Call Trace:
[328454.930490]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328454.950490]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328454.970490]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328454.987156]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328455.007157]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328455.027157]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328455.043823]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328519.904232] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328519.920483] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328520.027150] CPU 2:
[328520.033817] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328520.140483] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328520.167150] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328520.193817] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328520.210483] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328520.230483] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328520.253817] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328520.273817] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328520.293817] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328520.317150] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328520.340483] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328520.360483] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328520.380483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328520.403817] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328520.423817] Call Trace:
[328520.430483]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328520.450483]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328520.470483]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328520.487150]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328520.507150]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328520.527150]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328520.543817]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328585.400892] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328585.417144] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328585.523810] CPU 2:
[328585.530477] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328585.637143] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328585.663810] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328585.690477] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328585.707143] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328585.727143] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328585.750477] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328585.770477] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328585.790477] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328585.813811] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328585.837144] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328585.857143] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328585.877143] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328585.900477] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328585.920477] Call Trace:
[328585.927143]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328585.947143]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328585.967143]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328585.983810]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328586.003810]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328586.023810]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328586.040477]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328650.897552] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328650.913804] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328651.020470] CPU 2:
[328651.027137] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328651.133804] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328651.160470] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328651.187137] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328651.203803] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328651.223803] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328651.243803] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328651.267137] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328651.287137] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328651.310470] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328651.333804] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328651.353803] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328651.373803] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328651.397137] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328651.417137] Call Trace:
[328651.423803]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328651.443803]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328651.463803]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328651.480470]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328651.500470]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328651.520470]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328651.537137]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328716.397545] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328716.413797] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328716.520464] CPU 2:
[328716.527130] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328716.633797] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328716.660464] RIP: 0010:[<ffffffff8022f142>]  [<ffffffff8022f142>] 
flush_tlb_others_ipi+0x112/0x120
[328716.687130] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328716.700464] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328716.723797] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328716.743797] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328716.767130] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328716.787130] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328716.810464] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328716.833797] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328716.853797] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328716.873797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328716.897130] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328716.917130] Call Trace:
[328716.923797]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328716.943797]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328716.963797]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328716.980463]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328717.000464]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328717.020464]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328717.037130]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30
[328781.894206] BUG: soft lockup - CPU#2 stuck for 61s! [java:4933]
[328781.910457] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328782.017124] CPU 2:
[328782.023791] Modules linked in: sg forcedeth ipmi_watchdog sch_sfq 
ipt_ULOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_multiport 
iptable_filter ip_tables x_tables ohci_hcd ssb pcmcia pcmcia_core usbhid hid 
nf_conntrack_ftp nf_conntrack dm_mod ipmi_devintf ipmi_si ipmi_msghandler 
ehci_hcd usbcore amd74xx ide_core xfs exportfs sd_mod crc_t10dif aacraid 
scsi_mod [last unloaded: forcedeth]
[328782.130457] Pid: 4933, comm: java Tainted: G      D    
2.6.30.6_nogrsecurity-0.1 #1 H8DMU
[328782.157124] RIP: 0010:[<ffffffff8022f146>]  [<ffffffff8022f146>] 
flush_tlb_others_ipi+0x116/0x120
[328782.183790] RSP: 0000:ffff8802100e7d88  EFLAGS: 00000202
[328782.200457] RAX: 0000000000000080 RBX: 0000000000000080 RCX: 
0000000000000020
[328782.220457] RDX: 0000000000000081 RSI: 00000000000000f2 RDI: 
ffffffff807c2b98
[328782.243790] RBP: ffffffff8020c98e R08: ffff88021e8e90a8 R09: 
0000000000000000
[328782.263790] R10: 3880000000000000 R11: ffff880212870df8 R12: 
0000000000000000
[328782.283790] R13: ffffe20003ad1dc8 R14: ffffffff802af4ce R15: 
000000000000d580
[328782.307124] FS:  00007f3e462786f0(0000) GS:ffffc20000028000(0063) 
knlGS:00000000a73a4bb0
[328782.330457] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[328782.350457] CR2: 00000000f7f57168 CR3: 0000000210dd3000 CR4: 
00000000000006e0
[328782.370457] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[328782.393790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[328782.413791] Call Trace:
[328782.420457]  [<ffffffff8022f12f>] ? flush_tlb_others_ipi+0xff/0x120
[328782.440457]  [<ffffffff8022e6ea>] ? ptep_set_access_flags+0x1a/0x20
[328782.460457]  [<ffffffff802c6480>] ? do_wp_page+0xd0/0x840
[328782.477124]  [<ffffffff802ed4a1>] ? 
__mem_cgroup_commit_charge_swapin+0x1d1/0x250
[328782.497124]  [<ffffffff802c8790>] ? handle_mm_fault+0x660/0xb00
[328782.517124]  [<ffffffff80573f9b>] ? do_page_fault+0xfb/0x390
[328782.533790]  [<ffffffff80571bdf>] ? page_fault+0x1f/0x30

************************************************************************************
2.6.31 with debug options didn't work at all:

[   23.165601] Trying to unpack rootfs image as initramfs...
[   23.319343] BUG: unable to handle kernel NULL pointer dereference at 
(null)
[   23.320011] IP: [<(null)>] (null)
[   23.320011] PGD 0 
[   23.320011] Oops: 0010 [#1] SMP 
[   23.320011] last sysfs file: 
[   23.320011] CPU 0 
[   23.320011] Modules linked in:
[   23.320011] Pid: 1, comm: swapper Not tainted 2.6.31-0.testcrap.1 #1 
H8DMU
[   23.320011] RIP: 0010:[<0000000000000000>]  [<(null)>] (null)
[   23.320011] RSP: 0018:ffff88021ba13d18  EFLAGS: 00010246
[   23.320011] RAX: 0000000000008000 RBX: 0000000000000000 RCX: 
ffffffff00006558
[   23.320011] RDX: 0000000000006558 RSI: 0000000000004000 RDI: 
ffff880000e76000
[   23.320011] RBP: ffff88021ba13d90 R08: 0000000000000000 R09: 
0000000000000000
[   23.320011] R10: 0000000000006558 R11: 0000000000000000 R12: 
ffff88011b116960
[   23.320011] R13: ffff88021ace0000 R14: ffffffff81644ba2 R15: 
ffff880000e76000
[   23.320011] FS:  0000000000000000(0000) GS:ffffc90000000000(0000) 
knlGS:0000000000000000
[   23.320011] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[   23.320011] CR2: 0000000000000000 CR3: 00000000013e6000 CR4: 
00000000000006f0
[   23.320011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[   23.320011] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 
0000000000000400
[   23.320011] Process swapper (pid: 1, threadinfo ffff88021ba12000, task 
ffff88011ba90000)
[   23.320011] Stack:
[   23.320011]  ffffffff81664d01 ffff88021ba13d30 ffffffff00006558 
0000800000e76000
[   23.320011] <0> ffffffff816b22d8 0000000000000000 ffff880000e76000 
ffff880000e76000
[   23.320011] <0> 00000000811043e4 0000000000089bff 0000000000089bff 
ffff880000e76000
[   23.320011] Call Trace:
[   23.320011]  [<ffffffff81664d01>] ? gunzip+0x30a/0x42b
[   23.320011]  [<ffffffff81645067>] unpack_to_rootfs+0x22e/0x34b
[   23.320011]  [<ffffffff81644997>] ? error+0x0/0x17
[   23.320011]  [<ffffffff8164598e>] populate_rootfs+0x70/0x2eb
[   23.320011]  [<ffffffff81178000>] ? proc_create_data+0x70/0xc0
[   23.320011]  [<ffffffff8117c1d8>] ? proc_net_fops_create+0x18/0x20
[   23.320011]  [<ffffffff813cc419>] ? mutex_unlock+0x9/0x10
[   23.320011]  [<ffffffff812e0245>] ? register_pernet_subsys+0x35/0x40
[   23.320011]  [<ffffffff8164591e>] ? populate_rootfs+0x0/0x2eb
[   23.320011]  [<ffffffff81001047>] do_one_initcall+0x37/0x170
[   23.320011]  [<ffffffff81643137>] kernel_init+0x162/0x1bd
[   23.320011]  [<ffffffff81004f5a>] child_rip+0xa/0x20
[   23.320011]  [<ffffffff81642fd5>] ? kernel_init+0x0/0x1bd
[   23.320011]  [<ffffffff81004f50>] ? child_rip+0x0/0x20
[   23.320011] Code:  Bad RIP value.
[   23.320011] RIP  [<(null)>] (null)
[   23.320011]  RSP <ffff88021ba13d18>
[   23.320011] CR2: 0000000000000000
[   24.037117] ---[ end trace 9865ac5b5ee90b51 ]---
[   24.051068] Kernel panic - not syncing: Attempted to kill init!
[   24.068818] Pid: 1, comm: swapper Tainted: G      D    
2.6.31-0.testcrap.1 #1
[   24.090305] Call Trace:
[   24.097647]  [<ffffffff813ca8f4>] panic+0x73/0x125
[   24.112001]  [<ffffffff8104b0a4>] do_exit+0x744/0x830
[   24.127136]  [<ffffffff8104b1d5>] do_group_exit+0x45/0xb0
[   24.143408]  [<ffffffff813cebed>] oops_end+0x9d/0xe0
[   24.158277]  [<ffffffff810282aa>] no_context+0xea/0x260
[   24.173926]  [<ffffffff81028596>] __bad_area_nosemaphore+0x176/0x290
[   24.192961]  [<ffffffff8102dbc5>] ? kmemcheck_read+0x45/0x70
[   24.209994]  [<ffffffff8102e471>] ? kmemcheck_pte_lookup+0x11/0x60
[   24.228524]  [<ffffffff8102d821>] ? kmemcheck_show_addr+0x11/0x30
[   24.246776]  [<ffffffff8102d87b>] ? kmemcheck_show_all+0x3b/0x60
[   24.264768]  [<ffffffff81644ba2>] ? flush_buffer+0x0/0xa3
[   24.280991]  [<ffffffff810286be>] bad_area_nosemaphore+0xe/0x10
[   24.298721]  [<ffffffff813d051b>] do_page_fault+0x31b/0x3c0
[   24.315411]  [<ffffffff81644ba2>] ? flush_buffer+0x0/0xa3
[   24.331583]  [<ffffffff813cdfef>] page_fault+0x1f/0x30
[   24.347079]  [<ffffffff81644ba2>] ? flush_buffer+0x0/0xa3
[   24.363249]  [<ffffffff81664d01>] ? gunzip+0x30a/0x42b
[   24.378647]  [<ffffffff81645067>] unpack_to_rootfs+0x22e/0x34b
[   24.396483]  [<ffffffff81644997>] ? error+0x0/0x17
[   24.410851]  [<ffffffff8164598e>] populate_rootfs+0x70/0x2eb
[   24.427802]  [<ffffffff81178000>] ? proc_create_data+0x70/0xc0
[   24.445273]  [<ffffffff8117c1d8>] ? proc_net_fops_create+0x18/0x20
[   24.463890]  [<ffffffff813cc419>] ? mutex_unlock+0x9/0x10
[   24.480060]  [<ffffffff812e0245>] ? register_pernet_subsys+0x35/0x40
[   24.499090]  [<ffffffff8164591e>] ? populate_rootfs+0x0/0x2eb
[   24.516304]  [<ffffffff81001047>] do_one_initcall+0x37/0x170
[   24.533358]  [<ffffffff81643137>] kernel_init+0x162/0x1bd
[   24.549528]  [<ffffffff81004f5a>] child_rip+0xa/0x20
[   24.564400]  [<ffffffff81642fd5>] ? kernel_init+0x0/0x1bd
[   24.580570]  [<ffffffff81004f50>] ? child_rip+0x0/0x20
[   24.596041] Rebooting in 30 seconds..

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/