Re: [linux-pm] Kernel panic after resuming from S4 on Napa dual-core.

Re: [linux-pm] Kernel panic after resuming from S4 on Napa dual-core.

[Pavel Machek]

On Wed 12-07-06 09:29:40, Hu, Fenghua wrote:
> Hi all,
> 
> I am struggling with a S4 issue. On a Napa dual-core laptop, sometimes
> when resuming from S4, kernel panic occurs, then system reboots. After
> some investigation, I found it's due to CPU1(AP) can't be woken up in
> time, and
> this can be alleviated, even eliminated by introducing more delay in
> function wakeup_secondary_cpu().In official kernel tree, there were
> three lines in this function for delaying: mdelay(10), udelay(300) and
> udelay(200), now I changed them all to mdelay(500). Seems incorrect
> timing has led to the issue. Could you help me to root cause it? Thanks.

Do you have a copy of panic message and a patch? Or perhaps you just
want to ask on lkml for testing?

-- 
Thanks for all the (sleeping) penguins.

Re: Kernel panic after resuming from S4 on Napa dual-core.

[Hu, Fenghua]

Here is the message:

Linux version 2.6.16.9-9-s4-hufh (root@localhost.localdomain) (gcc version 3.4.3 20041212 (Asianux 2.0 3.4.3-9.EL4.2)) #24 SMP Thu Jul 27 11:36:08 CST 2006

BIOS-provided physical RAM map:

 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)

 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)

 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)

 BIOS-e820: 0000000000100000 - 000000001f7d0000 (usable)

 BIOS-e820: 000000001f7d0000 - 000000001f7e5600 (reserved)

 BIOS-e820: 000000001f7e5600 - 000000001f7f8000 (ACPI NVS)

 BIOS-e820: 000000001f7f8000 - 000000001f800000 (reserved)

 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)

 BIOS-e820: 00000000fed20000 - 00000000fed9b000 (reserved)

 BIOS-e820: 00000000feda0000 - 00000000fedc0000 (reserved)

 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)

 BIOS-e820: 00000000ffb00000 - 00000000ffc00000 (reserved)

 BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)

0MB HIGHMEM available.

503MB LOWMEM available.

DMI 2.4 present.

ACPI: PM-Timer IO Port: 0x1008

ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)

Processor #0 6:14 APIC version 20

ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)

Processor #1 6:14 APIC version 20

ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])

ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])

ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])

IOAPIC[0]: apic_id 1 already used, trying 2

IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23

ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)

ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)

Enabling APIC mode:  Flat.  Using 1 I/O APICs

More than 8 CPUs detected and CONFIG_X86_PC cannot handle it.

Use CONFIG_X86_GENERICARCH or CONFIG_X86_BIGSMP.

Using ACPI (MADT) for SMP configuration information

Allocating PCI resources starting at 20000000 (gap: 1f800000:df400000)

Built 1 zonelists

Kernel command line: ro root=LABEL=/ reboot=b resume=/dev/sda2 cpufreq.debug=7 cpufreq.debug_ratelimit=0 3 console=ttyS0,38400 console=tty0

Enabling fast FPU save and restore... done.

Enabling unmasked SIMD FPU exception support... done.

Initializing CPU#0

PID hash table entries: 2048 (order: 11, 32768 bytes)

Detected 1996.118 MHz processor.

Using pmtmr for high-res timesource

Console: colour VGA+ 80x25

Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)

Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)

Memory: 507172k/515904k available (1873k kernel code, 8232k reserved, 760k data, 224k init, 0k highmem)

Checking if this processor honours the WP bit even in supervisor mode... Ok.

Calibrating delay using timer specific routine.. 3998.31 BogoMIPS (lpj=1999157)

Mount-cache hash table entries: 512

monitor/mwait feature present.

using mwait in idle threads.

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 2048K

CPU: Physical Processor ID: 0

CPU: Processor Core ID: 0

Intel machine check architecture supported.

Intel machine check reporting enabled on CPU#0.

Checking 'hlt' instruction... OK.

ACPI: Looking for DSDT in initrd... not found!

 not found!

 tbxface-0109 [02] load_tables           : ACPI Tables successfully acquired

Parsing all Control Methods:

Table [DSDT](id 0007) - 1194 Objects with 107 Devices 354 Methods 46 Regions

Parsing all Control Methods:

Table [SSDT](id 0004) - 16 Objects with 4 Devices 5 Methods 0 Regions

Parsing all Control Methods:

Table [SSDT](id 0005) - 14 Objects with 0 Devices 5 Methods 0 Regions

ACPI Namespace successfully loaded at root c03fde70

evxfevnt-0091 [03] enable                : Transition to ACPI mode successful

CPU0: Intel Genuine Intel(R) CPU           T2500  @ 2.00GHz stepping 08

Booting processor 1/1 eip 3000

Initializing CPU#1

Calibrating delay using timer specific routine.. 3989.48 BogoMIPS (lpj=1994743)

monitor/mwait feature present.

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 2048K

CPU: Physical Processor ID: 0

CPU: Processor Core ID: 1

Intel machine check architecture supported.

Intel machine check reporting enabled on CPU#1.

CPU1: Intel Genuine Intel(R) CPU           T2500  @ 2.00GHz stepping 08

Total of 2 processors activated (7987.80 BogoMIPS).

ENABLING IO-APIC IRQs

..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1

checking TSC synchronization across 2 CPUs: passed.

Brought up 2 CPUs

migration_cost=1000

checking if image is initramfs... it is

Freeing initrd memory: 345k freed

NET: Registered protocol family 16

ACPI: bus type pci registered

PCI: PCI BIOS revision 2.10 entry at 0xf0322, last bus=32

PCI: Using MMCONFIG

ACPI: Subsystem revision 20060127

ACPI Error (evgpeblk-0284): Unknown GPE method type: C262 (name not of form _Lxx or _Exx) [20060127]

evgpeblk-0941 [06] ev_create_gpe_block   : GPE 00 to 1F [_GPE] 4 regs on int 0x9

evgpeblk-1037 [05] ev_initialize_gpe_bloc: Found 7 Wake, Enabled 5 Runtime GPEs in this block

Completing Region/Field/Buffer/Package initialization:..................................................................................................................................................................

Initialized 43/46 Regions 0/0 Fields 62/62 Buffers 57/69 Packages (1233 nodes)

Executing all Device _STA and_INI methods:........................................................................................................................

120 Devices found - executed 3 _STA, 6 _INI methods

ACPI: Interpreter enabled

ACPI: Using IOAPIC for interrupt routing

ACPI: PCI Root Bridge [C002] (0000:00)

ACPI: Assume root bridge [\_SB_.C002] bus is 0

PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1

PCI: Transparent bridge - 0000:00:1e.0

PCI: Bus #03 (-#06) is hidden behind transparent bridge #02 (-#03) (try 'pci=assign-busses')

Please report the result to linux-kernel to fix this permanently

ACPI: Embedded Controller [C005] (gpe 22) interrupt mode.

ACPI: Power Resource [C1E8] (on)

ACPI: Power Resource [C1F5] (on)

ACPI: Power Resource [C20B] (on)

ACPI: Power Resource [C213] (off)

ACPI: PCI Interrupt Link [C107] (IRQs 10 *11)

ACPI: PCI Interrupt Link [C108] (IRQs *10 11)

ACPI: PCI Interrupt Link [C109] (IRQs *10 11)

ACPI: PCI Interrupt Link [C10A] (IRQs *10 11)

ACPI: PCI Interrupt Link [C123] (IRQs *10 11)

ACPI: PCI Interrupt Link [C124] (IRQs 10 *11)

ACPI: PCI Interrupt Link [C125] (IRQs *10 11)

pci_link-0186 [03] pci_link_get_possible : Error evaluating _PRS

ACPI: Power Resource [C309] (off)

ACPI: Power Resource [C30A] (off)

ACPI: Power Resource [C30B] (off)

ACPI: Power Resource [C30C] (off)

Linux Plug and Play Support v0.97 (c) Adam Belay

pnp: PnP ACPI init

pnp: PnP ACPI: found 13 devices

usbcore: registered new driver usbfs

usbcore: registered new driver hub

PCI: Using ACPI for IRQ routing

PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report

pnp: 00:0b: ioport range 0x4d0-0x4d1 has been reserved

pnp: 00:0b: ioport range 0x1000-0x107f could not be reserved

pnp: 00:0b: ioport range 0x1100-0x113f has been reserved

pnp: 00:0b: ioport range 0x1200-0x121f has been reserved

PCI: Ignore bogus resource 6 [0:0] of 0000:00:02.0

PCI: Bridge: 0000:00:1c.0

  IO window: disabled.

  MEM window: e8000000-e80fffff

  PREFETCH window: disabled.

PCI: Bridge: 0000:00:1c.2

  IO window: 4000-5fff

  MEM window: e4000000-e7ffffff

  PREFETCH window: disabled.

PCI: Bridge: 0000:00:1c.3

  IO window: 2000-3fff

  MEM window: e0000000-e3ffffff

  PREFETCH window: disabled.

PCI: Bus 3, cardbus bridge: 0000:02:06.0

  IO window: 00007000-000070ff

  IO window: 00007400-000074ff

  PREFETCH window: 20000000-21ffffff

  MEM window: 22000000-23ffffff

PCI: Bridge: 0000:00:1e.0

  IO window: 7000-7fff

  MEM window: e8100000-e83fffff

  PREFETCH window: 20000000-21ffffff

acpi_bus-0201 [-3] bus_set_power         : Device is not power manageable

ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 16

acpi_bus-0201 [-3] bus_set_power         : Device is not power manageable

ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 18 (level, low) -> IRQ 17

acpi_bus-0201 [-3] bus_set_power         : Device is not power manageable

ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 19 (level, low) -> IRQ 18

acpi_bus-0201 [-3] bus_set_power         : Device is not power manageable

ACPI: PCI Interrupt 0000:02:06.0[A] -> GSI 18 (level, low) -> IRQ 17

Machine check exception polling timer started.

audit: initializing netlink socket (disabled)

audit(1154003309.702:1): initialized

Total HugeTLB memory allocated, 0

VFS: Disk quotas dquot_6.5.1

Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)

Initializing Cryptographic API

io scheduler noop registered

io scheduler anticipatory registered (default)

io scheduler deadline registered

io scheduler cfq registered

ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 16

assign_interrupt_mode Found MSI capability

ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 18 (level, low) -> IRQ 17

assign_interrupt_mode Found MSI capability

ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 19 (level, low) -> IRQ 18

assign_interrupt_mode Found MSI capability

Real Time Clock Driver v1.12ac

Linux agpgart interface v0.101 (c) Dave Jones

PNP: PS/2 Controller [PNP0303:C208,PNP0f13:C209] at 0x60,0x64 irq 1,12

serio: i8042 AUX port at 0x60,0x64 irq 12

serio: i8042 KBD port at 0x60,0x64 irq 1

Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

serial8250: ttyS0 at I/O 0x3f8 (irq = 0) is a 16550A

00:02: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize

Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2

ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

ICH7: IDE controller at PCI slot 0000:00:1f.1

ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 16 (level, low) -> IRQ 16

ICH7: chipset revision 1

ICH7: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0x60a0-0x60a7, BIOS settings: hda:pio, hdb:pio

ide-floppy driver 0.99.newide

usbmon: debugfs is not available

usbcore: registered new driver libusual

usbcore: registered new driver hiddev

usbcore: registered new driver usbhid

drivers/usb/input/hid-core.c: v2.6:USB HID core driver

mice: PS/2 mouse device common for all mice

md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

md: bitmap version 4.39

NET: Registered protocol family 2

input: AT Translated Set 2 keyboard as /class/input/input0

IP route cache hash table entries: 4096 (order: 2, 16384 bytes)

TCP established hash table entries: 16384 (order: 5, 131072 bytes)

TCP bind hash table entries: 16384 (order: 5, 131072 bytes)

TCP: Hash tables configured (established 16384 bind 16384)

TCP reno registered

TCP bic registered

NET: Registered protocol family 1

NET: Registered protocol family 17

Starting balanced_irq

Using IPI No-Shortcut mode

ACPI wakeup devices: 

C092 C0EA C0F1 C0F2 C0F3 C0F4 C0FB C21C C10E C21D C111 C21D 

ACPI: (supports S0 S3 S4 S5)

Freeing unused kernel memory: 224k freed

SCSI subsystem initialized

Driver 'sd' needs updating - please use bus_type methods

acpi_bus-0201 [-3] bus_set_power         : Device is not power manageable

ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 17 (level, low) -> IRQ 19

ahci 0000:00:1f.2: AHCI 0001.0100 32 slots 4 ports 1.5 Gbps 0x1 impl SATA mode

ahci 0000:00:1f.2: flags: 64bit ncq ilck stag pm led clo pmp pio slum part 

ata1: SATA max UDMA/133 cmd 0xE000A100 ctl 0x0 bmdma 0x0 irq 19

ata2: SATA max UDMA/133 cmd 0xE000A180 ctl 0x0 bmdma 0x0 irq 19

ata3: SATA max UDMA/133 cmd 0xE000A200 ctl 0x0 bmdma 0x0 irq 19

ata4: SATA max UDMA/133 cmd 0xE000A280 ctl 0x0 bmdma 0x0 irq 19

ata1: SATA link up 1.5 Gbps (SStatus 113)

ata1: dev 0 ATA-7, max UDMA/100, 78165360 sectors: LBA48

ata1: dev 0 configured for UDMA/100

scsi0 : ahci

ata2: SATA link down (SStatus 0)

scsi1 : ahci

ata3: SATA link down (SStatus 0)

scsi2 : ahci

ata4: SATA link down (SStatus 0)

scsi3 : ahci

  Vendor: ATA       Model: ST94813AS         Rev: 3.05

  Type:   Direct-Access                      ANSI SCSI revision: 05

SCSI device sda: 78165360 512-byte hdwr sectors (40021 MB)

sda: Write Protect is off

SCSI device sda: drive cache: write back

SCSI device sda: 78165360 512-byte hdwr sectors (40021 MB)

sda: Write Protect is off

SCSI device sda: drive cache: write back

 sda: sda1 sda2 sda3 sda4 < sda5 sda6 >

sd 0:0:0:0: Attached scsi disk sda

Attempting manual resume

Freezing cpus ...

CPU 1 is now offline

CPU1 is down

Shrinking memory...  \b-\bdone (0 pages freed)

Loading image metadata ... done (43 pages loaded)

Loading image data pages (43740 pages) ...     \b\b\b\b  0%\b\b\b\b  1%\b\b\b\b  2%\b\b\b\b  3%\b\b\b\b  4%\b\b\b\b  5%\b\b\b\b  6%\b\b\b\b  7%\b\b\b\b  8%\b\b\b\b  9%\b\b\b\b 10%\b\b\b\b 11%\b\b\b\b 12%\b\b\b\b 13%\b\b\b\b 14%\b\b\b\b 15%\b\b\b\b 16%\b\b\b\b 17%\b\b\b\b 18%\b\b\b\b 19%\b\b\b\b 20%\b\b\b\b 21%\b\b\b\b 22%\b\b\b\b 23%\b\b\b\b 24%\b\b\b\b 25%\b\b\b\b 26%\b\b\b\b 27%\b\b\b\b 28%\b\b\b\b 29%\b\b\b\b 30%\b\b\b\b 31%\b\b\b\b 32%\b\b\b\b 33%\b\b\b\b 34%\b\b\b\b 35%\b\b\b\b 36%\b\b\b\b 37%\b\b\b\b 38%\b\b\b\b 39%\b\b\b\b 40%\b\b\b\b 41%\b\b\b\b 42%\b\b\b\b 43%\b\b\b\b 44%\b\b\b\b 45%\b\b\b\b 46%\b\b\b\b 47%\b\b\b\b 48%\b\b\b\b 49%\b\b\b\b 50%\b\b\b\b 51%\b\b\b\b 52%\b\b\b\b 53%\b\b\b\b 54%\b\b\b\b 55%\b\b\b\b 56%\b\b\b\b 57%\b\b\b\b 58%\b\b\b\b 59%\b\b\b\b 60%\b\b\b\b 61%\b\b\b\b 62%\b\b\b\b 63%\b\b\b\b 64%\b\b\b\b 65%\b\b\b\b 66%\b\b\b\b 67%\b\b\b\b 68%\b\b\b\b 69%\b\b\b\b 70%\b\b\b\b 71%\b\b\b\b 72%\b\b\b\b 73%\b\b\b\b 74%\b\b\b\b 75%\b\b\b\b 76%\b\b\b\b 77%\b\b\b\b 78%\b\b\b\b 79%\b\b\b\b 80%\b\b\b\b 81%\b\b\b\b 82%\b\b\b\b 83%\b\b\b\b 84%\b\b\b\b 85%\b\b\b\b 86%\b\b\b\b 87%\b\b\b\b 88%\b\b\b\b 89%\b\b\b\b 90%\b\b\b\b 91%\b\b\b\b 92%\b\b\b\b 93%\b\b\b\b 94%\b\b\b\b 95%\b\b\b\b 96%\b\b\b\b 97%\b\b\b\b 98%\b\b\b\b 99%\b\b\b\b100%\b\b\b\bdone

pnp: Device 00:02 activated.

pnp: Device 00:03 activated.

pnp: Device 00:08 does not supported activation.

pnp: Device 00:09 does not supported activation.

ata1: enabling interrupts

ata1: resume device

ata1: SATA link up 1.5 Gbps (SStatus 113)

sata_get_dev_handle: SATA dev addr=0x1f0002, handle=0xc14aa3a8

ata1: dev 0 configured for UDMA/100

ipw3945: MAC is in deep sleep!

ipw3945: MAC is in deep sleep!

ipw3945: MAC is in deep sleep!

Thawing cpus ...

Booting processor 1/1 eip 3000

Not responding.

Inquiring remote APIC #1...

... APIC #1 ID: failed

... APIC #1 VERSION: failed

... APIC #1 SPIV: failed

Error taking cpu 1 up: -22

Kernel panic - not syncing: Not enough cpus

 <0>Rebooting in 10 seconds. 


Best regards
Hu, Fenghua


-----Original Message-----
From: Pavel Machek [mailto:pavel@suse.cz] 
Sent: 2006年7月15日 20:04
To: Hu, Fenghua
Cc: linux-acpi@vger.kernel.org; linux-pm@lists.osdl.org; lhcs-devel@lists.sourceforge.net
Subject: Re: [linux-pm] Kernel panic after resuming from S4 on Napa dual-core.

On Wed 12-07-06 09:29:40, Hu, Fenghua wrote:
> Hi all,
> 
> I am struggling with a S4 issue. On a Napa dual-core laptop, sometimes
> when resuming from S4, kernel panic occurs, then system reboots. After
> some investigation, I found it's due to CPU1(AP) can't be woken up in
> time, and
> this can be alleviated, even eliminated by introducing more delay in
> function wakeup_secondary_cpu().In official kernel tree, there were
> three lines in this function for delaying: mdelay(10), udelay(300) and
> udelay(200), now I changed them all to mdelay(500). Seems incorrect
> timing has led to the issue. Could you help me to root cause it? Thanks.

Do you have a copy of panic message and a patch? Or perhaps you just
want to ask on lkml for testing?

-- 
Thanks for all the (sleeping) penguins.

_______________________________________________
linux-pm mailing list
linux-pm@lists.osdl.org
https://lists.osdl.org/mailman/listinfo/linux-pm

Re: [linux-pm] Kernel panic after resuming from S4 on Napa dual-core.

[Rafael J. Wysocki]

On Wednesday 26 July 2006 07:45, Hu, Fenghua wrote:
> Here is the message:
> 
> Linux version 2.6.16.9-9-s4-hufh (root@localhost.localdomain) (gcc version 3.4.3 20041212 (Asianux 2.0 3.4.3-9.EL4.2)) #24 SMP Thu Jul 27 11:36:08 CST 2006
]--snip--[
> Thawing cpus ...
> Booting processor 1/1 eip 3000
> Not responding.
> Inquiring remote APIC #1...
> ... APIC #1 ID: failed
> ... APIC #1 VERSION: failed
> ... APIC #1 SPIV: failed
> Error taking cpu 1 up: -22
> Kernel panic - not syncing: Not enough cpus
>  <0>Rebooting in 10 seconds. 

This is a problem related to the CPU hotplug, but the panic is from
kernel/power/smp.c#enable_nonboot_cpus().

I don't think we should be panicking here.  It's better to just continue with
one CPU online IMHO.

Please apply the appended patch.  It should prevent the system from panicking,
but you will run on one CPU after resume.  Then, please check what's in the
file /sys/devices/system/cup/cpu1/online.  If 0, please do

# cat 1 >  /sys/devices/system/cup/cpu1/online

and see what happens.  If it doesn't go online, in which case
/sys/devices/system/cup/cpu1/online will contain 1, there should be some
error messages in dmesg.

Also after a fresh boot please try doing:

# cat 0 >  /sys/devices/system/cup/cpu1/online
# cat /proc/interrupts
(there should be no cpu1 in the output)
# cat 1 >  /sys/devices/system/cup/cpu1/online
# cat /proc/interrupts
(cpu1 should be present again)

If this doesn't work, CPU hotplug is totally broken on your system.
Otherwise, something in the suspend/resume code paths breaks it and we'll
have to figure out what it is.

Greetings,
Rafael


--
 kernel/power/smp.c |    1 -
 1 files changed, 1 deletion(-)

Index: linux-2.6.18-rc1-mm2/kernel/power/smp.c
===================================================================
--- linux-2.6.18-rc1-mm2.orig/kernel/power/smp.c
+++ linux-2.6.18-rc1-mm2/kernel/power/smp.c
@@ -55,7 +55,6 @@ void enable_nonboot_cpus(void)
 			continue;
 		}
 		printk("Error taking cpu %d up: %d\n", cpu, error);
-		panic("Not enough cpus");
 	}
 	cpus_clear(frozen_cpus);
 }