v2.6.22.1-rt3

v2.6.22.1-rt3

[Thomas Gleixner]

we are pleased to announce the v2.6.22.1-rt3 kernel

Attention! 

Ingo is off for a long weekend and therefor the download location for
this release is:

 http://www.tglx.de/projects/preempt-rt/2.6.22.1
  
more info about the -rt patchset can be found in the RT wiki:
  
   http://rt.wiki.kernel.org
 
This release is bugfix release:

- update of the x8664 -hrt queue (resolve boot problems)
- gtod vsyscall fix from Gregory Haskins

to build a 2.6.22.1-rt3 tree, the following patches should be applied:
 
   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.1.tar.bz2
   http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/patch-2.6.22.1-rt3.patch


	Thomas

Re: v2.6.22.1-rt3

[Remy Bohmer]

Thomas,

>  http://www.tglx.de/projects/preempt-rt/2.6.22.1

This is a dead link...
It should be: http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/

Remy

>
> more info about the -rt patchset can be found in the RT wiki:
>
>    http://rt.wiki.kernel.org
>
> This release is bugfix release:
>
> - update of the x8664 -hrt queue (resolve boot problems)
> - gtod vsyscall fix from Gregory Haskins
>
> to build a 2.6.22.1-rt3 tree, the following patches should be applied:
>
>    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.1.tar.bz2
>    http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/patch-2.6.22.1-rt3.patch
>
>
>         Thomas
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: v2.6.22.1-rt3

[Thomas Gleixner]

On Fri, 2007-07-13 at 13:36 +0200, Remy Bohmer wrote:
> Thomas,
> 
> >  http://www.tglx.de/projects/preempt-rt/2.6.22.1
> 
> This is a dead link...
> It should be: http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/

Grmbl, the publishing script choked.

	tglx

Re: v2.6.22.1-rt3

[Kevin Hilman]

Thomas,

In  arm-preempt-config.patch, the GENERIC_TIME is removed from the OMAP
arch.  Can you undo that removal?    OMAP is still GENERIC_TIME capable
unless something has been done to break it.

In other words, on top of -rt3:

Index: linux-2.6/arch/arm/Kconfig
===================================================================
--- linux-2.6.orig/arch/arm/Kconfig
+++ linux-2.6/arch/arm/Kconfig
@@ -394,6 +394,7 @@ config ARCH_DAVINCI
 config ARCH_OMAP
        bool "TI OMAP"
        select GENERIC_GPIO
+       select GENERIC_TIME
        help
          Support for TI's OMAP platform (OMAP1 and OMAP2).

Signed-off-by: Kevin Hilman <khilman@mvsita.com>

Re: v2.6.22.1-rt3

[David Bahi]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Fri, 2007-07-13 at 13:36 +0200, Remy Bohmer wrote:
> Thomas,
> 

> It should be: http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/
> 
> Remy
> 
> > This release is bugfix release:
> >
> > - update of the x8664 -hrt queue (resolve boot problems)
> > - gtod vsyscall fix from Gregory Haskins
> >

excellent. this definitely de-bricks -rt2 for my box.

still, there's this -rt1 acpi + spinlock panic if ! acpi=off 
(attached - now with some debug)

db



[-- Attachment #2: hardlock.txt --]
[-- Type: text/plain, Size: 26356 bytes --]

Linux version 2.6.22.1-rt3-debug (geeko@buildhost) (gcc version 4.1.2 20070115 (prerelease) (SUSE Linux)) #3 SMP PREEMPT RT Fri Jul7Command line: root=/dev/disk/by-id/scsi-SATA_HTS721010G9SA00_MPCZN7Y0H183WL-part6 vga=0x317 resume=/dev/sda5 splash=silent console=8BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
 BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000bfed0000 (usable)
 BIOS-e820: 00000000bfed0000 - 00000000bfedf000 (ACPI data)
 BIOS-e820: 00000000bfedf000 - 00000000bff00000 (ACPI NVS)
 BIOS-e820: 00000000bff00000 - 00000000c0000000 (reserved)
 BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
 BIOS-e820: 00000000fed14000 - 00000000fed1a000 (reserved)
 BIOS-e820: 00000000fed1c000 - 00000000fed90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
end_pfn_map = 1048576
DMI present.
ACPI: RSDP 000F6860, 0024 (r2 LENOVO)
ACPI: XSDT BFED1692, 008C (r1 LENOVO TP-79        2007  LTP        0)
ACPI: FACP BFED1800, 00F4 (r3 LENOVO TP-79        2007 LNVO        1)
ACPI Warning (tbfadt-0434): Optional field "Gpe1Block" has zero address or length: 000000000000102C/0 [20070126]
ACPI: DSDT BFED1C32, D1A1 (r1 LENOVO TP-79        2007 MSFT  100000E)
ACPI: FACS BFEF4000, 0040
ACPI: SSDT BFED19B4, 027E (r1 LENOVO TP-79        2007 MSFT  100000E)
ACPI: ECDT BFEDEDD3, 0052 (r1 LENOVO TP-79        2007 LNVO        1)
ACPI: TCPA BFEDEE25, 0032 (r2 LENOVO TP-79        2007 LNVO        1)
ACPI: APIC BFEDEE57, 0068 (r1 LENOVO TP-79        2007 LNVO        1)
ACPI: MCFG BFEDEEBF, 003C (r1 LENOVO TP-79        2007 LNVO        1)
ACPI: HPET BFEDEEFB, 0038 (r1 LENOVO TP-79        2007 LNVO        1)
ACPI: BOOT BFEDEFD8, 0028 (r1 LENOVO TP-79        2007  LTP        1)
ACPI: SSDT BFEF26D9, 025F (r1 LENOVO TP-79        2007 INTL 20050513)
ACPI: SSDT BFEF2938, 00A6 (r1 LENOVO TP-79        2007 INTL 20050513)
ACPI: SSDT BFEF29DE, 04F7 (r1 LENOVO TP-79        2007 INTL 20050513)
ACPI: SSDT BFEF2ED5, 08BD (r1 LENOVO TP-79        2007 INTL 20050513)
ACPI: SSDT BFEF3792, 069C (r1 LENOVO TP-79        2007 INTL 20050513)
No NUMA configuration found
Faking a node at 0000000000000000-00000000bfed0000
Bootmem setup node 0 0000000000000000-00000000bfed0000
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1048576
early_node_map[2] active PFN ranges
    0:        0 ->      159
    0:      256 ->   786128
ACPI: PM-Timer IO Port: 0x1008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Setting APIC routing to flat
ACPI: HPET id: 0x8086a201 base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at c4000000 (gap: c0000000:30000000)
PERCPU: Allocating 447432 bytes of per cpu data
Real-Time Preemption Support (C) 2004-2007 Ingo Molnar
Built 1 zonelists.  Total pages: 739589
Kernel command line: root=/dev/disk/by-id/scsi-SATA_HTS721010G9SA00_MPCZN7Y0H183WL-part6 vga=0x317 resume=/dev/sda5 splash=silent c8Initializing CPU#0
WARNING: experimental RCU implementation.
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
TSC calibrated against HPET
Marking TSC unstable due to TSCs unsynchronized
time.c: Detected 2327.494 MHz processor.
Console: colour dummy device 80x25
num_possible_cpus(): 2
CPU#0: allocated 3145728 bytes trace buffer.
CPU#0: allocated 3145728 bytes max-trace buffer.
CPU#1: allocated 3145728 bytes trace buffer.
CPU#1: allocated 3145728 bytes max-trace buffer.
allocated 6291456 bytes out-trace buffer.
tracer: a total of 18874368 bytes allocated.
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:      8
... MAX_LOCK_DEPTH:             30
... MAX_LOCKDEP_KEYS:         2048
... CLASSHASH_SIZE:           1024
... MAX_LOCKDEP_ENTRIES:     16384
... MAX_LOCKDEP_CHAINS:      32768
... CHAINHASH_SIZE:          16384
 memory used by lock dependency info: 2672 kB
 per task-struct memory footprint: 2160 bytes
------------------------
| Locking API testsuite:
----------------------------------------------------------------------------
                                 | spin |wlock |rlock |mutex | wsem | rsem |
  --------------------------------------------------------------------------
                     A-A deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
                 A-B-B-A deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
             A-B-B-C-C-A deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
             A-B-C-A-B-C deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
         A-B-B-C-C-D-D-A deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
         A-B-C-D-B-D-D-A deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
         A-B-C-D-B-C-D-A deadlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
                    double unlock:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
                  initialize held:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
                 bad unlock order:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
  --------------------------------------------------------------------------
              recursive read-lock:             |  ok  |             |  ok  |
           recursive read-lock #2:             |  ok  |             |  ok  |
            mixed read-write-lock:             |  ok  |             |  ok  |
            mixed write-read-lock:             |  ok  |             |  ok  |
  --------------------------------------------------------------------------
-------------------------------------------------------
Good, all  68 testcases passed! |
---------------------------------
Checking aperture...
Memory: 2930700k/3144512k available (2562k kernel code, 213424k reserved, 1460k data, 700k init)
Calibrating delay using timer specific routine.. 4660.34 BogoMIPS (lpj=2330172)
Security Framework v1.0.0 initialized
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
CPU 0/0 -> Node 0
using mwait in idle threads.
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
CPU0: Thermal monitoring enabled (TM2)
Freeing SMP alternatives: 20k freed
Unpacking initramfs... done
Freeing initrd memory: 6674k freed
ACPI: Core revision 20070126
Parsing all Control Methods:
Table [DSDT](id 0001) - 1623 Objects with 76 Devices 437 Methods 27 Regions
Parsing all Control Methods:
Table [SSDT](id 0002) - 11 Objects with 0 Devices 7 Methods 0 Regions
Parsing all Control Methods:
Table [SSDT](id 0003) - 7 Objects with 0 Devices 3 Methods 0 Regions
Parsing all Control Methods:
Table [SSDT](id 0004) - 4 Objects with 0 Devices 3 Methods 0 Regions
Parsing all Control Methods:
Table [SSDT](id 0005) - 14 Objects with 0 Devices 5 Methods 0 Regions
Parsing all Control Methods:
Table [SSDT](id 0006) - 48 Objects with 2 Devices 5 Methods 0 Regions
Parsing all Control Methods:
Table [SSDT](id 0007) - 47 Objects with 2 Devices 5 Methods 0 Regions
 tbxface-0587 [00] tb_load_namespace     : ACPI Tables successfully acquired
evxfevnt-0091 [00] enable                : Transition to ACPI mode successful
Using local APIC timer interrupts.
Detected 10.390 MHz APIC timer.
lockdep: not fixing up alternatives.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 4654.79 BogoMIPS (lpj=2327398)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
CPU 1/1 -> Node 0
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU1: Thermal monitoring enabled (TM2)
Intel(R) Core(TM)2 CPU         T7600  @ 2.33GHz stepping 06
Brought up 2 CPUs
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using MMCONFIG at f0000000 - f3ffffff
evgpeblk-0952 [00] ev_create_gpe_block   : GPE 00 to 1F [_GPE] 4 regs on int 0x9
evgpeblk-1048 [00] ev_initialize_gpe_bloc: Found 8 Wake, Enabled 2 Runtime GPEs in this block
Completing Region/Field/Buffer/Package initialization:..............................................................................Initialized 25/27 Regions 184/184 Fields 76/76 Buffers 43/55 Packages (1763 nodes)
Initializing Device/Processor/Thermal objects by executing _INI methods:........
Executed 8 _INI methods requiring 2 _STA executions (examined 86 objects)
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI quirk: region 1000-107f claimed by ICH6 ACPI/GPIO/TCO
PCI quirk: region 1180-11bf claimed by ICH6 GPIO
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
ACPI: Power Resource [PUBS] (on)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 14 devices
ACPI: ACPI bus type pnp unregistered
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
PCI-GART: No AMD northbridge found.
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
hpet0: 3 64-bit timers, 14318180 Hz
ACPI: RTC can wake from S4
pnp: 00:00: iomem range 0x0-0x9ffff could not be reserved
pnp: 00:00: iomem range 0xc0000-0xc3fff has been reserved
pnp: 00:00: iomem range 0xc4000-0xc7fff has been reserved
pnp: 00:00: iomem range 0xc8000-0xcbfff has been reserved
pnp: 00:02: iomem range 0xf0000000-0xf3ffffff could not be reserved
pnp: 00:02: iomem range 0xfed1c000-0xfed1ffff could not be reserved
pnp: 00:02: iomem range 0xfed14000-0xfed17fff could not be reserved
pnp: 00:02: iomem range 0xfed18000-0xfed18fff could not be reserved
PCI: Bridge: 0000:00:01.0
  IO window: 2000-2fff
  MEM window: ee100000-ee1fffff
  PREFETCH window: d0000000-dfffffff
PCI: Bridge: 0000:00:1c.0
  IO window: 3000-3fff
  MEM window: ee000000-ee0fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.1
  IO window: 4000-5fff
  MEM window: ec000000-edffffff
  PREFETCH window: e4000000-e40fffff
PCI: Bridge: 0000:00:1c.2
  IO window: 6000-7fff
  MEM window: e8000000-e9ffffff
  PREFETCH window: e4100000-e41fffff
PCI: Bridge: 0000:00:1c.3
  IO window: 8000-9fff
  MEM window: ea000000-ebffffff
  PREFETCH window: e4200000-e42fffff
PCI: Bus 22, cardbus bridge: 0000:15:00.0
  IO window: 0000a000-0000a0ff
  IO window: 0000a400-0000a4ff
  PREFETCH window: e0000000-e3ffffff
  MEM window: c4000000-c7ffffff
PCI: Bridge: 0000:00:1e.0
  IO window: a000-dfff
  MEM window: e4300000-e7ffffff
  PREFETCH window: e0000000-e3ffffff
ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 20 (level, low) -> IRQ 20
ACPI: PCI Interrupt 0000:00:1c.1[B] -> GSI 21 (level, low) -> IRQ 21
ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 22 (level, low) -> IRQ 22
ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 23 (level, low) -> IRQ 23
PCI: Enabling device 0000:00:1e.0 (0005 -> 0007)
ACPI: PCI Interrupt 0000:15:00.0[A] -> GSI 16 (level, low) -> IRQ 16
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 14, 104857600 bytes)
TCP bind hash table entries: 65536 (order: 11, 12582912 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
Simple Boot Flag at 0x35 set to 0x1
Initializing RT-Tester: OK
audit: initializing netlink socket (disabled)
audit(1184328004.428:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
vesafb: framebuffer at 0xd0000000, mapped to 0xffffc2000b800000, using 3072k, total 16384k
vesafb: mode is 1024x768x16, linelength=2048, pages=9
vesafb: scrolling: redraw
vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
Console: switching to colour frame buffer device 128x48
fb0: VESA VGA frame buffer device
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.102 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /class/input/input0
input: PC Speaker as /class/input/input1
IBM TrackPoint firmware: 0x0e, buttons: 3/3
input: TPPS/2 IBM TrackPoint as /class/input/input2
cpuidle: using governor ladder
NET: Registered protocol family 1
*****************************************************************************
*                                                                           *
*  REMINDER, the following debugging options are turned on in your .config: *
*                                                                           *
*        CONFIG_DEBUG_RT_MUTEXES                                            *
*        CONFIG_FUNCTION_TRACE                                              *
*        CONFIG_LOCKDEP                                                     *
*                                                                           *
*  they may increase runtime overhead and latencies.                        *
*                                                                           *
*****************************************************************************
Freeing unused kernel memory: 700k freed
Write protecting the kernel read-only data: 1040k
Starting udevd
Creating devices
Loading scsi_mod
SCSI subsystem initialized
Loading sd_mod
Loading libata
Loading ata_piix
ata_piix 0000:00:1f.2: MAP [ P0 P2 IDE IDE ]
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 16 (level, low) -> IRQ 16
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0x00000000000101f0 ctl 0x00000000000103f6 bmdma 0x00000000000118b0 irq 14
ata2: PATA max UDMA/100 cmd 0x0000000000010170 ctl 0x0000000000010376 bmdma 0x00000000000118b8 irq 15
ata1.00: ATA-7: HTS721010G9SA00, MCZIC10V, max UDMA/100
ata1.00: 195371568 sectors, multi 16: LBA48
ata1.00: configured for UDMA/100
ata2.00: ATAPI: MATSHITADVD-RAM UJ-842, RB01, max UDMA/33
ata2.00: configured for UDMA/33
scsi 0:0:0:0: Direct-Access     ATA      HTS721010G9SA00  MCZI PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 195371568 512-byte hardware sectors (100030 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 195371568 512-byte hardware sectors (100030 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 < sda5 sda6 sda7 sda8 sda9 >
sd 0:0:0:0: [sda] Attached SCSI disk
scsi 1:0:0:0: CD-ROM            MATSHITA DVD-RAM UJ-842   RB01 PQ: 0 ANSI: 5
Loading processor
sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
ACPI: SSDT BFEF1D36, 02C4 (r1  PmRef  Cpu0Ist      100 INTL 20050513)
Parsing all Control Methods:
Table [SSDT](id 0013) - 6 Objects with 0 Devices 4 Methods 0 Regions
ACPI: SSDT BFEF207F, 065A (r1  PmRef  Cpu0Cst      100 INTL 20050513)
Parsing all Control Methods:
Table [SSDT](id 0014) - 13 Objects with 0 Devices 1 Methods 0 Regions
Loading thermal
ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU0] (supports 8 throttling states)
ACPI: SSDT BFEF1C6E, 00C8 (r1  PmRef  Cpu1Ist      100 INTL 20050513)
Parsing all Control Methods:
Table [SSDT](id 0019) - 4 Objects with 0 Devices 4 Methods 0 Regions
ACPI: SSDT BFEF1FFA, 0085 (r1  PmRef  Cpu1Cst      100 INTL 20050513)
Loading fan
Parsing all Control Methods:
Table [SSDT](id 001A) - 1 Objects with 0 Devices 1 Methods 0 Regions
sd 0:0:0:0: Attached scsi generic sg0 type 0
sr 1:0:0:0: Attached scsi generic sg1 type 5
Loading edd
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU1] (supports 8 throttling states)
cpuidle: using driver acpi_idle
ACPI: Thermal Zone [THM0] (73 C)
ACPI: Thermal Zone [THM1] (76 C)
BIOS EDD facility v0.16 2004-Jun-25, 1 devices found
BUG: sleeping function called from invalid context swapper(0) at kernel/rtmutex.c:636
in_atomic():1 [00000001], irqs_disabled():1
1 lock held by swapper/0:
 #0:  (c3_lock){....}, at: [<ffffffff88093774>] acpi_idle_enter_c3+0x179/0x281 [processor]
irq event stamp: 58216
hardirqs last  enabled at (58215): [<ffffffff80259b9d>] trace_hardirqs_on+0xd/0x10
hardirqs last disabled at (58216): [<ffffffff80256ccd>] trace_hardirqs_off+0xd/0x10
softirqs last  enabled at (0): [<ffffffff80234223>] copy_process+0x513/0x1850
softirqs last disabled at (0): [<0000000000000000>] 0x0

Call Trace:
 [<ffffffff8022d803>] __might_sleep+0x113/0x150
 [<ffffffff80478305>] __rt_spin_lock+0x35/0x50
 [<ffffffff804784e8>] rt_spin_lock+0x58/0x70
 [<ffffffff88093774>] :processor:acpi_idle_enter_c3+0x179/0x281
 [<ffffffff803ed57a>] cpuidle_idle_call+0x9a/0xd0
 [<ffffffff803ed4e0>] cpuidle_idle_call+0x0/0xd0
 [<ffffffff80208d15>] cpu_idle+0x75/0xc0
 [<ffffffff8086a54d>] start_secondary+0x27d/0x370

---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<ffffffff80208d4c>] .... cpu_idle+0xac/0xc0
.....[<ffffffff8086a54d>] ..   ( <= start_secondary+0x27d/0x370)

BUG: scheduling with irqs disabled: swapper/0x00000001/0
stopped custom tracer.
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
 [<0000000000000000>]
PGD acac4067 PUD acad7067 PMD 0
Oops: 0010 [1] PREEMPT SMP
CPU 0
Modules linked in: edd fan thermal sg processor sr_mod cdrom ata_piix libata sd_mod scsi_mod
Pid: 0, comm: swapper Tainted: G      N 2.6.22.1-rt3-debug #3
RIP: 0010:[<0000000000000000>]  [<0000000000000000>]
RSP: 0018:ffffffff80857d40  EFLAGS: 00010003
RAX: ffffffff808515a0 RBX: ffff810037998740 RCX: 00000001da576082
RDX: 0000000000000001 RSI: ffff810037998740 RDI: ffff81000113da80
RBP: ffffffff80857d68 R08: 0000000000000000 R09: 0000000000000003
R10: 0000000000000000 R11: 0000000000000000 R12: 00000001da576082
R13: 0000000000000001 R14: ffff81000113da80 R15: ffff81000113da80
FS:  0000000000000000(0000) GS:ffffffff805ee280(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 00000000acae9000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffffffff80856000, task ffffffff80591cc0)
Stack:  ffffffff8022ad9c ffff81000113da80 ffff810037998740 00000001da576082
 0000000000000001 ffffffff80857d98 ffffffff8022c47b 000000000000001f
 0000000000000000 ffff810037998740 0000000000000001 ffffffff80857e58
Call Trace:
 [<ffffffff8022ad9c>] enqueue_task+0x4c/0x90
 [<ffffffff8022c47b>] activate_task+0x8b/0x110
 [<ffffffff8023049b>] try_to_wake_up+0x37b/0x570
 [<ffffffff802607d5>] wakeup_next_waiter+0x135/0x250
 [<ffffffff8023070a>] wake_up_process_mutex+0x1a/0x20
 [<ffffffff802607e4>] wakeup_next_waiter+0x144/0x250
 [<ffffffff8047761d>] rt_spin_lock_slowunlock+0x6d/0x90
 [<ffffffff8047840e>] rt_spin_unlock+0x2e/0x40
 [<ffffffff880937ad>] :processor:acpi_idle_enter_c3+0x1b2/0x281
 [<ffffffff803ed57a>] cpuidle_idle_call+0x9a/0xd0
 [<ffffffff803ed4e0>] cpuidle_idle_call+0x0/0xd0
 [<ffffffff80208d15>] cpu_idle+0x75/0xc0
 [<ffffffff8047535f>] rest_init+0x7f/0x90
 [<ffffffff8085f8ba>] start_kernel+0x2da/0x390
 [<ffffffff8085f119>] _sinittext+0x119/0x160

INFO: lockdep is turned off.
---------------------------
| preempt count: 00000003 ]
| 3-level deep critical section nesting:
----------------------------------------
.. [<ffffffff80208d4c>] .... cpu_idle+0xac/0xc0
.....[<ffffffff8047535f>] ..   ( <= rest_init+0x7f/0x90)
.. [<ffffffff8047935f>] .... __spin_lock_irqsave+0x2f/0x90
.....[<ffffffff804775ce>] ..   ( <= rt_spin_lock_slowunlock+0x1e/0x90)
.. [<ffffffff80478e79>] .... __spin_lock+0x19/0x70
.....[<ffffffff8022d8d6>] ..   ( <= task_rq_lock+0x56/0xa0)


Code:  Bad RIP value.
RIP  [<0000000000000000>]
 RSP <ffffffff80857d40>
CR2: 0000000000000000
Kernel panic - not syncing: Attempted to kill the idle task!

Call Trace:
 [<ffffffff802368e4>] panic+0xb4/0x190
 [<ffffffff80257fd7>] print_lock_contention_bug+0x27/0x100
 [<ffffffff80245c81>] __blocking_notifier_call_chain+0x21/0x80
 [<ffffffff80245cf6>] blocking_notifier_call_chain+0x16/0x20
 [<ffffffff8023afaa>] do_exit+0xa0a/0xa20
 [<ffffffff80477604>] rt_spin_lock_slowunlock+0x54/0x90
 [<ffffffff80236609>] do_oops_enter_exit+0x9/0xe0
 [<ffffffff8047c6dd>] do_page_fault+0x51d/0x920
 [<ffffffff80263789>] __module_text_address+0x9/0x70
 [<ffffffff8024ae27>] __kernel_text_address+0x27/0x40
 [<ffffffff8020b75d>] dump_trace+0xed/0x2b0
 [<ffffffff80212bdd>] save_stack_trace+0x2d/0x50
 [<ffffffff8047a46d>] error_exit+0x0/0x96
 [<ffffffff8022ad9c>] enqueue_task+0x4c/0x90
 [<ffffffff8022c47b>] activate_task+0x8b/0x110
 [<ffffffff8023049b>] try_to_wake_up+0x37b/0x570
 [<ffffffff802607d5>] wakeup_next_waiter+0x135/0x250
 [<ffffffff8023070a>] wake_up_process_mutex+0x1a/0x20
 [<ffffffff802607e4>] wakeup_next_waiter+0x144/0x250
 [<ffffffff8047761d>] rt_spin_lock_slowunlock+0x6d/0x90
 [<ffffffff8047840e>] rt_spin_unlock+0x2e/0x40
 [<ffffffff880937ad>] :processor:acpi_idle_enter_c3+0x1b2/0x281
 [<ffffffff803ed57a>] cpuidle_idle_call+0x9a/0xd0
 [<ffffffff803ed4e0>] cpuidle_idle_call+0x0/0xd0
 [<ffffffff80208d15>] cpu_idle+0x75/0xc0
 [<ffffffff8047535f>] rest_init+0x7f/0x90
 [<ffffffff8085f8ba>] start_kernel+0x2da/0x390
 [<ffffffff8085f119>] _sinittext+0x119/0x160

INFO: lockdep is turned off.
---------------------------
| preempt count: 00000004 ]
| 4-level deep critical section nesting:
----------------------------------------
.. [<ffffffff80208d4c>] .... cpu_idle+0xac/0xc0
.....[<ffffffff8047535f>] ..   ( <= rest_init+0x7f/0x90)
.. [<ffffffff8047935f>] .... __spin_lock_irqsave+0x2f/0x90
.....[<ffffffff804775ce>] ..   ( <= rt_spin_lock_slowunlock+0x1e/0x90)
.. [<ffffffff80478e79>] .... __spin_lock+0x19/0x70
.....[<ffffffff8022d8d6>] ..   ( <= task_rq_lock+0x56/0xa0)
.. [<ffffffff80236878>] .... panic+0x48/0x190
.....[<ffffffff8023afaa>] ..   ( <= do_exit+0xa0a/0xa20)

caller is rt_spin_lock_slowlock+0x99/0x220

Call Trace:
 [<ffffffff804766c1>] schedule+0x121/0x130
 [<ffffffff804777a9>] rt_spin_lock_slowlock+0x99/0x220
 [<ffffffff8047830d>] __rt_spin_lock+0x3d/0x50
 [<ffffffff804784e8>] rt_spin_lock+0x58/0x70
 [<ffffffff88093774>] :processor:acpi_idle_enter_c3+0x179/0x281
 [<ffffffff803ed57a>] cpuidle_idle_call+0x9a/0xd0
 [<ffffffff803ed4e0>] cpuidle_idle_call+0x0/0xd0
 [<ffffffff80208d15>] cpu_idle+0x75/0xc0
 [<ffffffff8086a54d>] start_secondary+0x27d/0x370

INFO: lockdep is turned off.
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<ffffffff80208d4c>] .... cpu_idle+0xac/0xc0
.....[<ffffffff8086a54d>] ..   ( <= start_secondary+0x27d/0x370)

BUG: scheduling while atomic: swapper/0x00000002/0, CPU#1
INFO: lockdep is turned off.
irq event stamp: 58288
hardirqs last  enabled at (58287): [<ffffffff80259b9d>] trace_hardirqs_on+0xd/0x10
hardirqs last disabled at (58288): [<ffffffff80256ccd>] trace_hardirqs_off+0xd/0x10
softirqs last  enabled at (0): [<ffffffff80234223>] copy_process+0x513/0x1850
softirqs last disabled at (0): [<0000000000000000>] 0x0

Call Trace:
 [<ffffffff80231004>] __schedule_bug+0x64/0x70
 [<ffffffff80475f61>] thread_return+0x5ea/0xac9
 [<ffffffff8020b990>] show_trace+0x70/0x80
 [<ffffffff8024a6b9>] pid_task+0x9/0x40
 [<ffffffff804765db>] schedule+0x3b/0x130
 [<ffffffff804777a9>] rt_spin_lock_slowlock+0x99/0x220
 [<ffffffff8047830d>] __rt_spin_lock+0x3d/0x50
 [<ffffffff804784e8>] rt_spin_lock+0x58/0x70
 [<ffffffff88093774>] :processor:acpi_idle_enter_c3+0x179/0x281
 [<ffffffff803ed57a>] cpuidle_idle_call+0x9a/0xd0
 [<ffffffff803ed4e0>] cpuidle_idle_call+0x0/0xd0
 [<ffffffff80208d15>] cpu_idle+0x75/0xc0
 [<ffffffff8086a54d>] start_secondary+0x27d/0x370

INFO: lockdep is turned off.
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<ffffffff80208d4c>] .... cpu_idle+0xac/0xc0
.....[<ffffffff8086a54d>] ..   ( <= start_secondary+0x27d/0x370)
.. [<ffffffff80475720>] .... __sched_text_start+0x20/0x277
.....[<ffffffff804765db>] ..   ( <= schedule+0x3b/0x130)


[-- Attachment #3: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: v2.6.22.1-rt3

[Daniel Walker]

On Fri, 2007-07-13 at 10:11 -0600, David Bahi wrote:
> On Fri, 2007-07-13 at 13:36 +0200, Remy Bohmer wrote:
> > Thomas,
> > 
> 
> > It should be: http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/
> > 
> > Remy
> > 
> > > This release is bugfix release:
> > >
> > > - update of the x8664 -hrt queue (resolve boot problems)
> > > - gtod vsyscall fix from Gregory Haskins
> > >
> 
> excellent. this definitely de-bricks -rt2 for my box.
> 
> still, there's this -rt1 acpi + spinlock panic if ! acpi=off 
> (attached - now with some debug)
> 
> db
> 

Below is an only compile tested patch to drop the spinlocks out of the
c3 enter idle function, and use atomics instead. It looks related to
your panic.. I'm in the process of testing it, but you more than welcome
to try it..

---
 drivers/acpi/processor_idle.c |   13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

Index: linux-2.6.22.1/drivers/acpi/processor_idle.c
===================================================================
--- linux-2.6.22.1.orig/drivers/acpi/processor_idle.c
+++ linux-2.6.22.1/drivers/acpi/processor_idle.c
@@ -950,8 +950,7 @@ static int acpi_idle_enter_c2(struct cpu
 	return ticks_elapsed_in_us(t1, t2);
 }
 
-static int c3_cpu_count;
-static DEFINE_SPINLOCK(c3_lock);
+static atomic_t c3_cpu_count;
 
 /**
  * acpi_idle_enter_c3 - enters an ACPI C3 state-type
@@ -990,16 +989,13 @@ static int acpi_idle_enter_c3(struct cpu
 
 	/* disable bus master */
 	if (pr->flags.bm_check) {
-		spin_lock(&c3_lock);
-			c3_cpu_count++;
-		if (c3_cpu_count == num_online_cpus()) {
+		if (atomic_inc_return(&c3_cpu_count) == num_online_cpus()) {
 			/*
 			 * All CPUs are trying to go to C3
 			 * Disable bus master arbitration
 			 */
 			acpi_set_register(ACPI_BITREG_ARB_DISABLE, 1);
 		}
-		spin_unlock(&c3_lock);
 	} else {
 		/* SMP with no shared cache... Invalidate cache  */
 		ACPI_FLUSH_CPU_CACHE();
@@ -1012,12 +1008,9 @@ static int acpi_idle_enter_c3(struct cpu
 	t2 = inl(acpi_gbl_FADT.xpm_timer_block.address);
 
 	if (pr->flags.bm_check) {
-		spin_lock(&c3_lock);
 		/* Enable bus master arbitration */
-		if (c3_cpu_count == num_online_cpus())
+		if (atomic_dec_return(&c3_cpu_count) == (num_online_cpus() - 1))
 			acpi_set_register(ACPI_BITREG_ARB_DISABLE, 0);
-		c3_cpu_count--;
-		spin_unlock(&c3_lock);
 	}
 
 #ifdef CONFIG_GENERIC_TIME

Re: v2.6.22.1-rt3

[Kevin Hilman]

Thomas,

A typo in preempt-irqs-core.patch, where IRQF_TIMER is changed to
_IRQF_TIMER but called later as __IRQF_TIMER.

Here's a patch to compile, but not sure if you want one or two underscores.

With these two paches, -rt3 is building/booting for ARM/OMAP1.

Kevin

Index: linux-2.6/include/linux/interrupt.h
===================================================================
--- linux-2.6.orig/include/linux/interrupt.h
+++ linux-2.6/include/linux/interrupt.h
@@ -52,7 +52,7 @@
 #define IRQF_SAMPLE_RANDOM     0x00000040
 #define IRQF_SHARED            0x00000080
 #define IRQF_PROBE_SHARED      0x00000100
-#define _IRQF_TIMER            0x00000200
+#define __IRQF_TIMER           0x00000200
 #define IRQF_PERCPU            0x00000400
 #define IRQF_NOBALANCING       0x00000800
 #define IRQF_IRQPOLL           0x00001000

Re: v2.6.22.1-rt3 - Early INT13 boot crash

[Carsten Emde]

Thomas,

> we are pleased to announce the v2.6.22.1-rt3 kernel
Thanks a lot!

Using the .config file from the previous stable version (2.6.21.6-rt21), 
the new version runs okay on an x86_64 system. On an i386 system, 
however, the system is crashing at an early boot stage displaying an 
INT13 reg dump at the window bottom.

After disabling paravirtualization support (CONFIG_PARAVIRT), the new 
version is working fine on i386 as well.

Just in case someone out there is experiencing the same problem.

	--cbe

Re: v2.6.22.1-rt3

[Fernando Lopez-Lezcano]

On Fri, 2007-07-13 at 13:22 +0200, Thomas Gleixner wrote:
> we are pleased to announce the v2.6.22.1-rt3 kernel
> 
> Attention! 
> 
> Ingo is off for a long weekend and therefor the download location for
> this release is:
> 
>  http://www.tglx.de/projects/preempt-rt/2.6.22.1
>   
> more info about the -rt patchset can be found in the RT wiki:
>   
>    http://rt.wiki.kernel.org
>  
> This release is bugfix release:
> 
> - update of the x8664 -hrt queue (resolve boot problems)
> - gtod vsyscall fix from Gregory Haskins

Same problem as reported yesterday in 2.6.22.1-rt2 in a T61 laptop, boot
hangs, last BUG printed is similar to this (numbers changed since
yesterday, of course, functions listed appear to be the same). No serial
port available to dump everything...

This was copied from the screen yesterday:

BUG: spinlock lockup on CPU#1, swapper/0, c318da88
[<c0405f34>] show_trace_log_lvl+0x1a/0x2f
[<c0406a09>] show_trace+-x12/0x14
[<c0406a71>] dump_stack+0x16/0x18
[<c0617a91>] _raw_spin_lock+0xc1/0xe2
[<c061743f>] __spin_lock_irq+0x14/0x16
[<c061541d>] __sched_tex_start+0xd5/0xaef
[<c061600e>] schedule+0xe0/0xfa
[<c0616c15>] rt_spin_lock_slowlock+0xcf/0x14f
[<c061724b>] __rt_spin_lock+0x3d/0x40
[<c0617256>] rt_spin_lock+0x8/0xa
[<c052f95c>] acpi_idle_enter_c3+0x12d/0x232
[<c059af51>] cpuidle_idle_call+0x56/0x79
[<c04033a5>] cpu_idle+0x9d/0xda
[<c0419e21>] start_secondary+0x34e/0x356
[<00000000>] 0x0

Same .config as before.
-- Fernando

Re: v2.6.22.1-rt3

[David Bahi]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Fri, 2007-07-13 at 09:29 -0700, Daniel Walker wrote:
> On Fri, 2007-07-13 at 10:11 -0600, David Bahi wrote:
> > On Fri, 2007-07-13 at 13:36 +0200, Remy Bohmer wrote:
> > > Thomas,
> > > 
> > 
> > > It should be: http://www.tglx.de/projects/preempt-rt/2.6.22.1-rt3/
> > > 

> Below is an only compile tested patch to drop the spinlocks out of the
> c3 enter idle function, and use atomics instead. It looks related to
> your panic.. I'm in the process of testing it, but you more than welcome
> to try it..

outstanding!

2.6.22.1-rt3-rt #1 SMP PREEMPT RT Fri Jul 13 16:08:12 EDT 2007 \
 x86_64 x86_64 x86_64 GNU/Linux

thank you very much - this is a confirmed fix for my t60p laptop. 

cyclictest results ran as i typed this:

./cyclictest -p80 -t2 -n
0.15 0.04 0.01 3/217 5011

T: 0 ( 4963) P:80 I:1000 C: 168698 Min:      4 Act:   67 Avg:   33 Max:
832
T: 1 ( 4964) P:79 I:1500 C: 112466 Min:      4 Act:   50 Avg:   34 Max:
825



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: v2.6.22.1-rt3

[Josh Triplett]

On Fri, 2007-07-13 at 13:22 +0200, Thomas Gleixner wrote:
> we are pleased to announce the v2.6.22.1-rt3 kernel
[...]
> This release is bugfix release:
> 
> - update of the x8664 -hrt queue (resolve boot problems)
> - gtod vsyscall fix from Gregory Haskins

I can confirm that this patch fixes booting on an 8-CPU x86-64 box that
-rt2 would not boot on.

- Josh Triplett

2.6.22.1-rt3 lockups

[Rui Nuno Capela]

Hi,

Current 2.6.22.1-rt3 is locking-up on any of my x86 SMP machines, in
very rare and non-deterministic occasions and normal desktop workloads,
but seems to be more probable when high disk I/O is underway.

At least, I was able to capture some crash traces, via serial console,
with nmi_watchdog=1.

...
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: appletalk ax25 ipx p8023 snd_rtctimer snd_seq_dummy
snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event w83627hf
hwmon_vid snd_seq button hwmon battery ac eeprom loop dm_mod ohci1394
ieee1394 wacom usbhid hid ff_memless snd_ice1712 snd_ice17xx_ak4xxx
snd_ak4xxx_adda snd_cs8427 snd_i2c firewire_ohci snd_mpu401_uart sk98lin
firewire_core ide_cd nvidia(P) snd_cs46xx gameport snd_rawmidi
snd_seq_device cdrom crc_itu_t snd_intel8x0 snd_ac97_codec ehci_hcd
uhci_hcd ac97_bus snd_pcm intel_agp snd_timer i2c_i801 agpgart snd
i2c_core soundcore snd_page_alloc shpchp iTCO_wdt rtc_cmos usbcore
pci_hotplug rtc_core rtc_lib ext3 mbcache jbd edd fan piix thermal
processor ide_disk ide_core
CPU:    1
EIP:    0060:[<00000000>]    Tainted: P       VLI
EFLAGS: 00210007   (2.6.22.1-rt3.0 #1)
EIP is at _stext+0x3feff000/0x20
eax: c1812a80   ebx: c03bb540   ecx: 00000001   edx: c038e3c0
esi: c038e3c0   edi: 00000001   ebp: c5099d6c   esp: c5099d50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000003
Process cc1plus (pid: 20669, ti=c5099000 task=deca8070 task.ti=c5099000)
Stack: c011a8fc 3302bd39 000009cb c1812a80 c1812a80 3302bd39 000009cb
c5099d90
       c011b45f 3302bd39 000009cb 00000001 c038e3c0 00000001 00000000
c038e3c0
       c5099df4 c011e04d c5099dfc c011ddeb 00000000 0000001f c1812a80
0000001f
Call Trace:
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
 [<c0106521>] show_registers+0x201/0x330
 [<c0106768>] die+0x118/0x260
 [<c03041e3>] do_page_fault+0x193/0x600
 [<c03028fa>] error_code+0x72/0x78
 [<c011b45f>] activate_task+0x4f/0xb0
 [<c011e04d>] try_to_wake_up+0x2bd/0x420
 [<c011e229>] wake_up_process_mutex+0x19/0x20
 [<c014257c>] wakeup_next_waiter+0xec/0x1a0
 [<c03016ec>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301fa6>] rt_spin_unlock+0x26/0x30
 [<c015b394>] put_zone_pcp+0x14/0x20
 [<c015c215>] get_page_from_freelist+0x145/0x380
 [<c015c4a4>] __alloc_pages+0x54/0x2d0
 [<c016526d>] __handle_mm_fault+0x7dd/0x9a0
 [<c0304348>] do_page_fault+0x2f8/0x600
 [<c03028fa>] error_code+0x72/0x78
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feff000/0x20 SS:ESP 0068:c5099d50
__sched_text_start+0x91e/0xbd0
 [<c030086e>] schedule+0x2e/0x110
 [<c030184e>] rt_spin_lock_slowlock+0x8e/0x170
 [<c0301fd0>] __rt_spin_lock+0x20/0x30
 [<c0301fe8>] rt_spin_lock+0x8/0x10
 [<c015b53b>] get_zone_pcp+0x2b/0x50
 [<c015be97>] free_hot_cold_page+0xc7/0x190
 [<c015bfba>] free_hot_page+0xa/0x10
 [<c015bfe7>] __free_pages+0x27/0x30
 [<c015c016>] free_pages+0x26/0x30
 [<c01765e5>] quicklist_trim+0xc5/0x110
 [<c011875e>] check_pgt_cache+0x1e/0x20
 [<c01033b9>] cpu_idle+0x49/0xb0
 [<c02ff88d>] rest_init+0x6d/0x70
 [<c03c1e01>] start_kernel+0x301/0x3b0
 [<00000000>] _stext+0x3feff000/0x20
 =======================
NMI watchdog detected lockup on CPU#1 (5000/5000)

...
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: appletalk ax25 ipx p8023 snd_rtctimer snd_seq_dummy
snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq
w83627hf hwmon_vid button battery hwmon eeprom ac loop dm_mod ohci1394
ieee1394 wacom usbhid snd_ice1712 hid snd_ice17xx_ak4xxx snd_cs46xx
snd_ak4xxx_adda ff_memless snd_cs8427 gameport sk98lin firewire_ohci
snd_i2c snd_mpu401_uart nvidia(P) snd_rawmidi firewire_core
snd_seq_device crc_itu_t snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm
snd_timer ide_cd snd iTCO_wdt soundcore shpchp cdrom snd_page_alloc
pci_hotplug i2c_i801 i2c_core ehci_hcd uhci_hcd usbcore rtc_cmos
rtc_core rtc_lib intel_agp agpgart ext3 mbcache jbd edd fan piix thermal
processor ide_disk ide_core
CPU:    1
EIP:    0060:[<00000000>]    Tainted: P       VLI
EFLAGS: 00010003   (2.6.22.1-rt3.0 #1)
EIP is at _stext+0x3feff000/0x20
eax: c181ca80   ebx: c03bb540   ecx: 00000001   edx: dfca0c30
esi: dfca0c30   edi: 00000001   ebp: d2025b54   esp: d2025b38
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000003
Process rsync (pid: 18436, ti=d2025000 task=ca152730 task.ti=d2025000)
Stack: c011a8fc 00387b36 000008de c181ca80 c181ca80 00387b36 000008de
d2025b78
       c011b45f 00387b36 000008de 00000001 dfca0c30 00000001 00000001
dfca0c30
       d2025bdc c011e04d d2025be4 c011ddeb 00000000 0000001f c181ca80
0000001f
Call Trace:
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
 [<c0106521>] show_registers+0x201/0x330
 [<c0106768>] die+0x118/0x260
 [<c03041e3>] do_page_fault+0x193/0x600
 [<c03028fa>] error_code+0x72/0x78
 [<c011b45f>] activate_task+0x4f/0xb0
 [<c011e04d>] try_to_wake_up+0x2bd/0x420
 [<c011e229>] wake_up_process_mutex+0x19/0x20
 [<c014257c>] wakeup_next_waiter+0xec/0x1a0
 [<c03016ec>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301fa6>] rt_spin_unlock+0x26/0x30
 [<c015b394>] put_zone_pcp+0x14/0x20
 [<c015c215>] get_page_from_freelist+0x145/0x380
 [<c015c4a4>] __alloc_pages+0x54/0x2d0
 [<c0174e79>] cache_alloc_refill+0x2b9/0x510
 [<c0174bae>] kmem_cache_alloc+0x7e/0x90
 [<f95817a2>] ext3_alloc_inode+0x12/0x50 [ext3]
 [<c018c609>] alloc_inode+0x19/0x190
 [<c018c7ce>] iget_locked+0x4e/0x140
 [<f9581470>] ext3_lookup+0x80/0xe0 [ext3]
 [<c017ffd8>] do_lookup+0x138/0x180
 [<c018219d>] __link_path_walk+0x81d/0xe10
 [<c01827d6>] link_path_walk+0x46/0xd0
 [<c0182879>] path_walk+0x19/0x20
 [<c0182a2b>] do_path_lookup+0x7b/0x220
 [<c0183458>] __user_walk_fd+0x38/0x50
 [<c017be2e>] vfs_lstat_fd+0x1e/0x50
 [<c017bea1>] vfs_lstat+0x11/0x20
 [<c017bec4>] sys_lstat64+0x14/0x30
 [<c01051d2>] sysenter_past_esp+0x5f/0x85
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feff000/0x20 SS:ESP 0068:d2025b38
NMI watchdog detected lockup on CPU#1 (5000/5000)

Pid: 18436, comm:                rsync
EIP: 0060:[<c03022b6>] CPU: 1
EIP is at __spin_lock+0x16/0x20
 EFLAGS: 00000082    Tainted: P        (2.6.22.1-rt3.0 #1)
EAX: c181ca80 EBX: c181ca80 ECX: dfc2c1b0 EDX: d2025000
ESI: c0403a80 EDI: dfc2c1b0 EBP: d202598c DS: 007b ES: 007b FS: 00d8
CR0: 8005003b CR2: ffffffd5 CR3: 1027e000 CR4: 000006d0
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c0106e12>] show_trace+0x12/0x20
 [<c0103af3>] show_regs+0x183/0x190
 [<c0303420>] nmi_watchdog_tick+0x1f0/0x290
 [<c0302e57>] do_nmi+0x77/0x260
 [<c03029a3>] nmi_stack_correct+0x26/0x2b
 [<c011bb77>] task_rq_lock+0x37/0x70
 [<c011e00a>] try_to_wake_up+0x27a/0x420
 [<c011e1c8>] default_wake_function+0x18/0x20
 [<c013689b>] autoremove_wake_function+0x1b/0x50
 [<c011a6a9>] __wake_up_common+0x39/0x60
 [<c01204a3>] __wake_up+0x33/0x60
 [<c012374b>] wake_up_klogd+0x3b/0x40
 [<c01ee247>] bust_spinlocks+0x27/0x30
 [<c01067bc>] die+0x16c/0x260
 [<c03041e3>] do_page_fault+0x193/0x600
 [<c03028fa>] error_code+0x72/0x78
 [<c011b45f>] activate_task+0x4f/0xb0
 [<c011e04d>] try_to_wake_up+0x2bd/0x420
 [<c011e229>] wake_up_process_mutex+0x19/0x20
 [<c014257c>] wakeup_next_waiter+0xec/0x1a0
 [<c03016ec>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301fa6>] rt_spin_unlock+0x26/0x30
 [<c015b394>] put_zone_pcp+0x14/0x20
 [<c015c215>] get_page_from_freelist+0x145/0x380
 [<c015c4a4>] __alloc_pages+0x54/0x2d0
 [<c0174e79>] cache_alloc_refill+0x2b9/0x510
 [<c0174bae>] kmem_cache_alloc+0x7e/0x90
 [<f95817a2>] ext3_alloc_inode+0x12/0x50 [ext3]
 [<c018c609>] alloc_inode+0x19/0x190
 [<c018c7ce>] iget_locked+0x4e/0x140
 [<f9581470>] ext3_lookup+0x80/0xe0 [ext3]
 [<c017ffd8>] do_lookup+0x138/0x180
 [<c018219d>] __link_path_walk+0x81d/0xe10
 [<c01827d6>] link_path_walk+0x46/0xd0
 [<c0182879>] path_walk+0x19/0x20
 [<c0182a2b>] do_path_lookup+0x7b/0x220
 [<c0183458>] __user_walk_fd+0x38/0x50
 [<c017be2e>] vfs_lstat_fd+0x1e/0x50
 [<c017bea1>] vfs_lstat+0x11/0x20
 [<c017bec4>] sys_lstat64+0x14/0x30
 [<c01051d2>] sysenter_past_esp+0x5f/0x85
 =======================
NMI watchdog detected lockup on CPU#0 (0/5000)

...
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: appletalk ax25 ipx p8023 snd_rtctimer snd_seq_dummy
snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq button
battery ac w83627hf hwmon_vid hwmon eeprom loop dm_mod wacom usbhid hid
ff_memless nvidia(P) snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_cs46xx sk98lin snd_i2c gameport snd_mpu401_uart
snd_rawmidi snd_seq_device ohci1394 ieee1394 snd_intel8x0 snd_ac97_codec
ac97_bus snd_pcm firewire_ohci firewire_core snd_timer crc_itu_t ide_cd
cdrom shpchp intel_agp snd i2c_i801 iTCO_wdt agpgart pci_hotplug
i2c_core soundcore ehci_hcd snd_page_alloc uhci_hcd usbcore rtc_cmos
rtc_core rtc_lib ext3 mbcache jbd edd fan piix thermal processor
ide_disk ide_core
CPU:    0
EIP:    0060:[<00000000>]    Tainted: P       VLI
EFLAGS: 00213003   (2.6.22.1-rt3.0 #1)
EIP is at _stext+0x3feff000/0x20
eax: c1812a80   ebx: c03bb540   ecx: 00000001   edx: c038e3c0
esi: c038e3c0   edi: 00000001   ebp: f4fb0d6c   esp: f4fb0d50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000003
Process Xorg (pid: 4145, ti=f4fb0000 task=dfd8c6b0 task.ti=f4fb0000)
Stack: c011a8fc f00c393c 00000b0a c1812a80 c1812a80 f00c393c 00000b0a
f4fb0d90
       c011b45f f00c393c 00000b0a 00000001 c038e3c0 00000000 00000000
c038e3c0
       f4fb0df4 c011e04d 00000000 c180d000 00000000 0000001f c1812a80
f4fb0e20
Call Trace:
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
 [<c0106521>] show_registers+0x201/0x330
 [<c0106768>] die+0x118/0x260
 [<c03041e3>] do_page_fault+0x193/0x600
 [<c03028fa>] error_code+0x72/0x78
 [<c011b45f>] activate_task+0x4f/0xb0
 [<c011e04d>] try_to_wake_up+0x2bd/0x420
 [<c011e229>] wake_up_process_mutex+0x19/0x20
 [<c014257c>] wakeup_next_waiter+0xec/0x1a0
 [<c03016ec>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301fa6>] rt_spin_unlock+0x26/0x30
 [<c015b394>] put_zone_pcp+0x14/0x20
 [<c015c215>] get_page_from_freelist+0x145/0x380
 [<c015c51f>] __alloc_pages+0xcf/0x2d0
 [<c016526d>] __handle_mm_fault+0x7dd/0x9a0
 [<c0304348>] do_page_fault+0x2f8/0x600
 [<c03028fa>] error_code+0x72/0x78
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feff000/0x20 SS:ESP 0068:f4fb0d50
NMI watchdog detected lockup on CPU#1 (5000/5000)

Pid: 2779, comm:                klogd
EIP: 0060:[<c03022b9>] CPU: 1
EIP is at __spin_lock+0x19/0x20
 EFLAGS: 00000082    Tainted: P        (2.6.22.1-rt3.0 #1)
EAX: c1812a80 EBX: c1812a80 ECX: 00000001 EDX: f4d11000
ESI: c0403a80 EDI: dff1c1b0 EBP: f4d11d1c DS: 007b ES: 007b FS: 00d8
CR0: 8005003b CR2: b7faa000 CR3: 1fe08000 CR4: 000006d0
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c0106e12>] show_trace+0x12/0x20
 [<c0103af3>] show_regs+0x183/0x190
 [<c0303420>] nmi_watchdog_tick+0x1f0/0x290
 [<c0302e57>] do_nmi+0x77/0x260
 [<c03029a3>] nmi_stack_correct+0x26/0x2b
 [<c011bb77>] task_rq_lock+0x37/0x70
 [<c011ddb7>] try_to_wake_up+0x27/0x420
 [<c011e1c8>] default_wake_function+0x18/0x20
 [<c011a6a9>] __wake_up_common+0x39/0x60
 [<c012050b>] __wake_up_sync+0x3b/0x50
 [<c02939b9>] sock_def_readable+0x79/0x80
 [<c02fafc0>] unix_dgram_sendmsg+0x450/0x500
 [<c028eff4>] sock_aio_write+0x114/0x130
 [<c0178160>] do_sync_write+0xd0/0x110
 [<c0178a5d>] vfs_write+0x14d/0x160
 [<c017907d>] sys_write+0x3d/0x70
 [<c01051d2>] sysenter_past_esp+0x5f/0x85
 =======================
NMI watchdog detected lockup on CPU#0 (0/5000)

...

Here are the complete console captures:

  http://www.rncbc.org/datahub/console-2.6.22.1-rt3.0-1.log
  http://www.rncbc.org/datahub/console-2.6.22.1-rt3.0-2.log
  http://www.rncbc.org/datahub/console-2.6.22.1-rt3.0-3.log

.config evidence:

  http://www.rncbc.org/datahub/config-2.6.22.1-rt3.0

Cheers.
--
rncbc aka Rui Nuno Capela

Re: v2.6.22.1-rt3

[Daniel Walker]

I reworked the broken out series for 2.6.22.1-rt5 (note not -rt3) so
that it's a little more bisectable. I found that many of the patches
would compile but wouldn't boot. 

Combined patch,
ftp://source.mvista.com/pub/dwalker/rt/patch-2.6.22.1-rt4-dw1


The broken out series is here,
ftp://source.mvista.com/pub/dwalker/rt/patch-2.6.22.1-rt4-dw1.tar.gz

Below is a diff between the 2.6.22.1-rt4 series and mine, and a
interdiff between the two combined patches.

--- patches-2.6.22.1-rt4/series	2007-07-16 02:29:51.000000000 -0700
+++ patches/series	2007-07-19 20:40:00.000000000 -0700
@@ -306,18 +306,21 @@
 #
 # IRQ threading
 #
+preempt-softirqs-core.patch
 preempt-irqs-core.patch
+preempt-irqs-softirq-in-hardirq.patch
+preempt-irqs-direct-debug-keyboard.patch
 preempt-irqs-timer.patch
 preempt-irqs-hrtimer.patch
 
 preempt-irqs-i386.patch
+preempt-irqs-i386-ioapic-mask-quirk.patch
 
 preempt-irqs-mips.patch
 
 preempt-irqs-x86-64.patch
 preempt-irqs-x86-64-ioapic-mask-quirk.patch
 
-preempt-irqs-i386-ioapic-mask-quirk.patch
 preempt-irqs-arm.patch
 preempt-irqs-arm-fix-oprofile.patch
 
@@ -352,7 +355,7 @@
 rt-mutex-mips.patch
 
 rt-mutex-ppc.patch
-rt-mtex-ppc-fix-a5.patch
+rt-mutex-ppc-fix-a5.patch
 
 rt-mutex-x86-64.patch
 
@@ -402,6 +405,7 @@
 #
 # Posix-cpu-timers in a thread
 #
+preempt-realtime-warn-and-bug-on.patch
 cputimer-thread-rt_A0.patch
 cputimer-thread-rt-fix.patch
 posix-cpu-timers-fix.patch
@@ -501,7 +505,6 @@
 preempt-realtime-timer.patch
 preempt-realtime-usb.patch
 
-preempt-realtime-warn-and-bug-on.patch
 preempt-realtime-warn-and-bug-on-fix.patch
 
 #
@@ -611,7 +614,6 @@
 # Softirq modifications
 #
 new-softirq-code.patch
-new-softirq-code-fixlets.patch
 softirq-per-cpu-assumptions-fixes.patch
 smp-processor-id-fixups.patch
 fix-migrating-softirq.patch
@@ -659,13 +661,9 @@
 #
 # not yet backmerged tail patches:
 #
-hrt-rt-fix-merge-artifact.patch
 preempt-rt-no-slub.patch
 rfkill-input-fix.patch
-fork.c-takeover-tasklets-warning-fix.patch
 
 paravirt-function-pointer-fix.patch
-hpet-build-fix.patch
-rtc.c-build-fix.patch
 version.patch
 

diff -u linux/arch/i386/kernel/hpet.c linux-2.6.22.1/arch/i386/kernel/hpet.c
--- linux/arch/i386/kernel/hpet.c
+++ linux-2.6.22.1/arch/i386/kernel/hpet.c	2007-07-20 02:22:56.000000000 +0000
@@ -9,7 +9,6 @@
 #include <linux/pm.h>
 
 #include <asm/fixmap.h>
-#include <asm/i8253.h>
 #include <asm/hpet.h>
 #include <asm/i8253.h>
 #include <asm/io.h>
diff -u linux/include/asm-generic/bug.h linux-2.6.22.1/include/asm-generic/bug.h
--- linux/include/asm-generic/bug.h
+++ linux-2.6.22.1/include/asm-generic/bug.h	2007-07-20 03:15:15.000000000 +0000
@@ -94,14 +94,2 @@
 
-#ifdef CONFIG_PREEMPT_RT
-# define BUG_ON_RT(c)			BUG_ON(c)
-# define BUG_ON_NONRT(c)		do { } while (0)
-# define WARN_ON_RT(condition)		WARN_ON(condition)
-# define WARN_ON_NONRT(condition)	do { } while (0)
-#else
-# define BUG_ON_RT(c)			do { } while (0)
-# define BUG_ON_NONRT(c)		BUG_ON(c)
-# define WARN_ON_RT(condition)		do { } while (0)
-# define WARN_ON_NONRT(condition)	WARN_ON(condition)
-#endif
-
 #endif
diff -u linux/kernel/softirq.c linux-2.6.22.1/kernel/softirq.c
--- linux/kernel/softirq.c
+++ linux-2.6.22.1/kernel/softirq.c	2007-07-20 03:15:17.000000000 +0000
@@ -102,7 +102,6 @@
 
 	if (unlikely(!tsk))
 		return;
-#if 1
 #if defined(CONFIG_PREEMPT_SOFTIRQS) && defined(CONFIG_PREEMPT_HARDIRQS)
 	/*
 	 * Optimization: if we are in a hardirq thread context, and
@@ -117,7 +116,6 @@
 			(tsk->normal_prio == current->normal_prio))
 		return;
 #endif
-#endif
 	/*
 	 * Wake up the softirq task:
 	 */

Re: v2.6.22.1-rt3

[Daniel Walker]

On Thu, 2007-07-19 at 20:37 -0700, Daniel Walker wrote:
> I reworked the broken out series for 2.6.22.1-rt5 (note not -rt3) so

Woops , I mean , 2.6.22.1-rt4 here.. 

Re: v2.6.22.1-rt3

[Thomas Gleixner]

On Thu, 2007-07-19 at 20:37 -0700, Daniel Walker wrote:
> The broken out series is here,
> ftp://source.mvista.com/pub/dwalker/rt/patch-2.6.22.1-rt4-dw1.tar.gz

I'll pick that up soon.

Thanks,

	tglx

2.6.22.1-rt4 lockups

[Rui Nuno Capela]

Hi,

As with -rt3, I was able to capture one more crash trace, via serial
console, with nmi_watchdog=1.

Yes, current 2.6.22.1-rt4 is still locking-up on my ix86 SMT/SMP boxes.
I'll have to wait for some hours of uptime and normal desktop use and
then it just locks-up without warning.

Last couple of occurrences were all while browsing with firefox
(2.0.0.5) or using openoffice.org (2.0.4) but in rare and
non-deterministic fashion I must say.

It looks very similar to the previous ones I've reported before for
-rt3, but I am no expert in these things.

...
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: tun appletalk ax25 ipx p8023 snd_rtctimer
snd_seq_dummy snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event
snd_seq w83627hf hwmon_vid hwmon eeprom button battery ac loop dm_mod
wacom usbhid hid ff_memless ohci1394 ieee1394 nvidia(P) snd_cs46xx
gameport firewire_ohci snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda
snd_cs8427 snd_i2c snd_mpu401_uart snd_rawmidi snd_seq_device
firewire_core sk98lin snd_intel8x0 crc_itu_t snd_ac97_codec ac97_bus
ide_cd snd_pcm cdrom snd_timer uhci_hcd ehci_hcd i2c_i801 snd rtc_cmos
shpchp iTCO_wdt i2c_core usbcore rtc_core pci_hotplug soundcore
intel_agp rtc_lib agpgart snd_page_alloc ext3 mbcache jbd edd fan piix
thermal processor ide_disk ide_core
CPU:    0
EIP:    0060:[<00000000>]    Tainted: P       VLI
EFLAGS: 00213006   (2.6.22.1-rt4.0 #1)
EIP is at _stext+0x3feff000/0x20
eax: c1812a80   ebx: c03bb540   ecx: 00000001   edx: c038e3c0
esi: c038e3c0   edi: 00000001   ebp: f6fe1d6c   esp: f6fe1d50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000003
Process Xorg (pid: 4101, ti=f6fe1000 task=f754ec30 task.ti=f6fe1000)
Stack: c011a94c 04882eab 00000ca9 c1812a80 c1812a80 04882eab 00000ca9
f6fe1d90
       c011b4af 04882eab 00000ca9 00000001 c038e3c0 00000000 00000000
c038e3c0
       f6fe1df4 c011e09d f6fe1dfc c011de3b 00000000 0000001f c1812a80
0000001f
Call Trace:
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
 [<c0106521>] show_registers+0x201/0x330
 [<c0106768>] die+0x118/0x260
 [<c0304233>] do_page_fault+0x193/0x600
 [<c030294a>] error_code+0x72/0x78
 [<c011b4af>] activate_task+0x4f/0xb0
 [<c011e09d>] try_to_wake_up+0x2bd/0x420
 [<c011e279>] wake_up_process_mutex+0x19/0x20
 [<c01425cc>] wakeup_next_waiter+0xec/0x1a0
 [<c030173c>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301ff6>] rt_spin_unlock+0x26/0x30
 [<c015b3e4>] put_zone_pcp+0x14/0x20
 [<c015c265>] get_page_from_freelist+0x145/0x380
 [<c015c4f4>] __alloc_pages+0x54/0x2d0
 [<c01652bd>] __handle_mm_fault+0x7dd/0x9a0
 [<c0304398>] do_page_fault+0x2f8/0x600
 [<c030294a>] error_code+0x72/0x78
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feff000/0x20 SS:ESP 0068:f6fe1d50
NMI watchdog detected lockup on CPU#1 (5000/5000)
...


Complete serial console capture:

  http://www.rncbc.org/datahub/console-2.6.22.1-rt4.0-1.log

.config evidence:

  http://www.rncbc.org/datahub/config-2.6.22.1-rt4.0

Cheers.
--
rncbc aka Rui Nuno Capela

Re: 2.6.22.1-rt4 lockups

[Rui Nuno Capela]

Hi again,

Sorry to bother, but got another one. Please advise whether these dumps
are any useful or are just garbage.

...
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: appletalk ax25 ipx p8023 snd_rtctimer snd_seq_dummy
snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq
w83627hf hwmon_vid hwmon button eeprom battery ac loop dm_mod ohci1394
ieee1394 snd_ice1712 snd_ice17xx_ak4xxx snd_ak4xxx_adda snd_cs8427
nvidia(P) snd_i2c wacom usbhid hid ff_memless firewire_ohci snd_cs46xx
snd_mpu401_uart gameport snd_rawmidi firewire_core crc_itu_t
snd_seq_device sk98lin snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm
snd_timer snd intel_agp soundcore ide_cd ehci_hcd uhci_hcd cdrom
iTCO_wdt shpchp snd_page_alloc agpgart usbcore i2c_i801 pci_hotplug
i2c_core rtc_cmos rtc_core rtc_lib ext3 mbcache jbd edd fan piix thermal
processor ide_disk ide_core
CPU:    0
EIP:    0060:[<00000000>]    Tainted: P       VLI
EFLAGS: 00010003   (2.6.22.1-rt4.0 #1)
EIP is at _stext+0x3feff000/0x20
eax: c1812a80   ebx: c03bb540   ecx: 00000001   edx: c038e3c0
esi: c038e3c0   edi: 00000001   ebp: c64a9d6c   esp: c64a9d50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068  preempt:00000003
Process thunderbird-bin (pid: 12848, ti=c64a9000 task=e4514db0
task.ti=c64a9000)
Stack: c011a94c d1027dbe 00001dc1 c1812a80 c1812a80 d1027dbe 00001dc1
c64a9d90
       c011b4af d1027dbe 00001dc1 00000001 c038e3c0 00000000 00000000
c038e3c0
       c64a9df4 c011e09d 00000000 00000000 00000000 0000001f c1812a80
c64a9e20
Call Trace:
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
 [<c0106521>] show_registers+0x201/0x330
 [<c0106768>] die+0x118/0x260
 [<c0304233>] do_page_fault+0x193/0x600
 [<c030294a>] error_code+0x72/0x78
 [<c011b4af>] activate_task+0x4f/0xb0
 [<c011e09d>] try_to_wake_up+0x2bd/0x420
 [<c011e279>] wake_up_process_mutex+0x19/0x20
 [<c01425cc>] wakeup_next_waiter+0xec/0x1a0
 [<c030173c>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301ff6>] rt_spin_unlock+0x26/0x30
 [<c015b3e4>] put_zone_pcp+0x14/0x20
 [<c015c265>] get_page_from_freelist+0x145/0x380
 [<c015c4f4>] __alloc_pages+0x54/0x2d0
 [<c01652bd>] __handle_mm_fault+0x7dd/0x9a0
 [<c0304398>] do_page_fault+0x2f8/0x600
 [<c030294a>] error_code+0x72/0x78
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] _stext+0x3feff000/0x20 SS:ESP 0068:c64a9d50
NMI watchdog detected lockup on CPU#1 (5000/5000)

Pid: 2882, comm:                klogd
EIP: 0060:[<c0302309>] CPU: 1
EIP is at __spin_lock+0x19/0x20
 EFLAGS: 00000082    Tainted: P        (2.6.22.1-rt4.0 #1)
EAX: c1812a80 EBX: c1812a80 ECX: 00000001 EDX: f6c57000
ESI: c0403a80 EDI: dff03230 EBP: f6c57d1c DS: 007b ES: 007b FS: 00d8
CR0: 8005003b CR2: ae41a000 CR3: 3730e000 CR4: 000006d0
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c0106e12>] show_trace+0x12/0x20
 [<c0103af3>] show_regs+0x183/0x190
 [<c0303470>] nmi_watchdog_tick+0x1f0/0x290
 [<c0302ea7>] do_nmi+0x77/0x260
 [<c03029f3>] nmi_stack_correct+0x26/0x2b
 [<c011bbc7>] task_rq_lock+0x37/0x70
 [<c011de07>] try_to_wake_up+0x27/0x420
 [<c011e218>] default_wake_function+0x18/0x20
 [<c011a6f9>] __wake_up_common+0x39/0x60
 [<c012055b>] __wake_up_sync+0x3b/0x50
 [<c0293a09>] sock_def_readable+0x79/0x80
 [<c02fb010>] unix_dgram_sendmsg+0x450/0x500
 [<c028f044>] sock_aio_write+0x114/0x130
 [<c01781b0>] do_sync_write+0xd0/0x110
 [<c0178aad>] vfs_write+0x14d/0x160
 [<c01790cd>] sys_write+0x3d/0x70
 [<c01051d2>] sysenter_past_esp+0x5f/0x85
 =======================
NMI watchdog detected lockup on CPU#0 (0/5000)

Pid: 12848, comm:      thunderbird-bin
EIP: 0060:[<c0302309>] CPU: 0
EIP is at __spin_lock+0x19/0x20
 EFLAGS: 00000082    Tainted: P        (2.6.22.1-rt4.0 #1)
EAX: c1812a80 EBX: c1812a80 ECX: 00000000 EDX: c040d000
ESI: c0403a80 EDI: f74cf8f0 EBP: c040df50 DS: 007b ES: 007b FS: 00d8
CR0: 8005003b CR2: ffffffd5 CR3: 0c391000 CR4: 000006d0
 [<c010622a>] show_trace_log_lvl+0x1a/0x30
 [<c0106e12>] show_trace+0x12/0x20
 [<c0103af3>] show_regs+0x183/0x190
 [<c0303470>] nmi_watchdog_tick+0x1f0/0x290
 [<c0302ea7>] do_nmi+0x77/0x260
 [<c03029f3>] nmi_stack_correct+0x26/0x2b
 [<c011bbc7>] task_rq_lock+0x37/0x70
 [<c011de07>] try_to_wake_up+0x27/0x420
 [<c011e2b9>] wake_up_process+0x19/0x20
 [<c01519c7>] redirect_hardirq+0x47/0x60
 [<c015343b>] handle_fasteoi_irq+0x6b/0x100
 [<c01075f4>] do_IRQ+0x94/0x100
 [<c0105beb>] common_interrupt+0x23/0x28
 [<c0126798>] do_exit+0x88/0x890
 [<c01068a7>] die+0x257/0x260
 [<c0304233>] do_page_fault+0x193/0x600
 [<c030294a>] error_code+0x72/0x78
 [<c011b4af>] activate_task+0x4f/0xb0
 [<c011e09d>] try_to_wake_up+0x2bd/0x420
 [<c011e279>] wake_up_process_mutex+0x19/0x20
 [<c01425cc>] wakeup_next_waiter+0xec/0x1a0
 [<c030173c>] rt_spin_lock_slowunlock+0x4c/0x70
 [<c0301ff6>] rt_spin_unlock+0x26/0x30
 [<c015b3e4>] put_zone_pcp+0x14/0x20
 [<c015c265>] get_page_from_freelist+0x145/0x380
 [<c015c4f4>] __alloc_pages+0x54/0x2d0
 [<c01652bd>] __handle_mm_fault+0x7dd/0x9a0
 [<c0304398>] do_page_fault+0x2f8/0x600
 [<c030294a>] error_code+0x72/0x78
 =======================
...


Complete serial console capture:

  http://www.rncbc.org/datahub/console-2.6.22.1-rt4.0-2.log

.config evidence:

  http://www.rncbc.org/datahub/config-2.6.22.1-rt4.0

Bye now
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: 2.6.22.1-rt4 lockups

[Daniel Walker]

On Sat, 2007-07-21 at 23:07 +0100, Rui Nuno Capela wrote:

> Call Trace:
>  [<c010622a>] show_trace_log_lvl+0x1a/0x30
>  [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
>  [<c0106521>] show_registers+0x201/0x330
>  [<c0106768>] die+0x118/0x260
>  [<c0304233>] do_page_fault+0x193/0x600
>  [<c030294a>] error_code+0x72/0x78
>  [<c011b4af>] activate_task+0x4f/0xb0
>  [<c011e09d>] try_to_wake_up+0x2bd/0x420
>  [<c011e279>] wake_up_process_mutex+0x19/0x20
>  [<c01425cc>] wakeup_next_waiter+0xec/0x1a0
>  [<c030173c>] rt_spin_lock_slowunlock+0x4c/0x70
>  [<c0301ff6>] rt_spin_unlock+0x26/0x30
>  [<c015b3e4>] put_zone_pcp+0x14/0x20
>  [<c015c265>] get_page_from_freelist+0x145/0x380
>  [<c015c4f4>] __alloc_pages+0x54/0x2d0
>  [<c01652bd>] __handle_mm_fault+0x7dd/0x9a0
>  [<c0304398>] do_page_fault+0x2f8/0x600
>  [<c030294a>] error_code+0x72/0x78
>  =======================

I was able to reproduce a similar looking hang when I combine kernbench
running with another load (I used ltpstress.sh from LTP) ..

I'm debugging it now ..

Daniel

Re: 2.6.22.1-rt4 lockups

[Daniel Walker]

On Mon, 2007-07-23 at 09:08 -0700, Daniel Walker wrote:
> On Sat, 2007-07-21 at 23:07 +0100, Rui Nuno Capela wrote:
> 
> > Call Trace:
> >  [<c010622a>] show_trace_log_lvl+0x1a/0x30
> >  [<c01062f6>] show_stack_log_lvl+0xb6/0xe0
> >  [<c0106521>] show_registers+0x201/0x330
> >  [<c0106768>] die+0x118/0x260
> >  [<c0304233>] do_page_fault+0x193/0x600
> >  [<c030294a>] error_code+0x72/0x78
> >  [<c011b4af>] activate_task+0x4f/0xb0
> >  [<c011e09d>] try_to_wake_up+0x2bd/0x420
> >  [<c011e279>] wake_up_process_mutex+0x19/0x20
> >  [<c01425cc>] wakeup_next_waiter+0xec/0x1a0
> >  [<c030173c>] rt_spin_lock_slowunlock+0x4c/0x70
> >  [<c0301ff6>] rt_spin_unlock+0x26/0x30
> >  [<c015b3e4>] put_zone_pcp+0x14/0x20
> >  [<c015c265>] get_page_from_freelist+0x145/0x380
> >  [<c015c4f4>] __alloc_pages+0x54/0x2d0
> >  [<c01652bd>] __handle_mm_fault+0x7dd/0x9a0
> >  [<c0304398>] do_page_fault+0x2f8/0x600
> >  [<c030294a>] error_code+0x72/0x78
> >  =======================
> 
> I was able to reproduce a similar looking hang when I combine kernbench
> running with another load (I used ltpstress.sh from LTP) ..
> 
> I'm debugging it now ..

It looks like sched_class->enqueue_task() is NULL and that's why the
system hangs ..

The reason why that happens is because check_pgt_cache() is called from
the idle thread, and with PREEMPT_RT check_pgt_cache() locks at least
one mutex .. Once the idle thread is on a wait_list, as soon as it's
woke by the mutex owner the system will crash in enqueue_task. Since the
idle thread has a NULL sched_class->enqueue_task ..

check_pgt_cache() is already getting called from the desched_thread() ,
so I think it could just be removed from i386 cpu_idle().

Anyone have comments on the theory above?

Daniel

Re: 2.6.22.1-rt4 lockups

[Ingo Molnar]

* Daniel Walker <dwalker@mvista.com> wrote:

> It looks like sched_class->enqueue_task() is NULL and that's why the 
> system hangs ..
> 
> The reason why that happens is because check_pgt_cache() is called 
> from the idle thread, and with PREEMPT_RT check_pgt_cache() locks at 
> least one mutex .. Once the idle thread is on a wait_list, as soon as 
> it's woke by the mutex owner the system will crash in enqueue_task. 
> Since the idle thread has a NULL sched_class->enqueue_task ..
> 
> check_pgt_cache() is already getting called from the desched_thread() 
> , so I think it could just be removed from i386 cpu_idle().
> 
> Anyone have comments on the theory above?

yeah, that call definitely looks wrong in cpu_idle(). Most of the other 
check_pgd_cache() calls introduced by commit f1d1a842 look wrong too in 
an -rt context. Fix is below.

	Ingo

Index: linux-rt.q/arch/i386/kernel/process.c
===================================================================
--- linux-rt.q.orig/arch/i386/kernel/process.c
+++ linux-rt.q/arch/i386/kernel/process.c
@@ -189,7 +189,6 @@ void cpu_idle(void)
 
 			tick_nohz_stop_sched_tick();
 
-			check_pgt_cache();
 			rmb();
 			idle = pm_idle;