PC Engines APU panics on boot since 4.2.4

PC Engines APU panics on boot since 4.2.4

[Marc Haber]

Hi,

I run a PC Engines APU in my basement for local infrastructure with
KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the
usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors
activated (3999.89 BogoMIPS)" message. The message I would expect
instead of the kernel panic would be "devtmpfs: initialized", as one
can see on a successful 4.2.3 boot with identical kernel configuration.

Loading Linux 4.2.5-zgsrv20080 ...
Loading initial ramdisk ...
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 4.2.5-zgsrv20080 (mh@fan) (gcc version 5.2.1 20151010 (Debian 5.2.1-22) ) #1 SMP PREEMPT Wed Oct 28 18:14:25 UTC 2015
Command line: BOOT_IMAGE=/vmlinuz-4.2.5-zgsrv20080 root=/dev/mapper/aida-aida_root ro console=ttyS0,115200n8 net.ifnames=1
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
x86/fpu: Legacy x87 FPU detected.
x86/fpu: Using 'lazy' FPU context switches.
e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000df14dfff] usable
BIOS-e820: [mem 0x00000000df14e000-0x00000000dfffffff] reserved
BIOS-e820: [mem 0x00000000f8000000-0x00000000f8ffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000011effffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.7 present.
e820: last_pfn = 0x11f000 max_arch_pfn = 0x400000000
x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT
e820: last_pfn = 0xdf14e max_arch_pfn = 0x400000000
found SMP MP-table at [mem 0x000f25b0-0x000f25bf] mapped at [ffff8800000f25b0]
Using GB pages for direct mapping
init_memory_mapping: [mem 0x00000000-0x000fffff]
init_memory_mapping: [mem 0x11ee00000-0x11effffff]
init_memory_mapping: [mem 0x100000000-0x11edfffff]
init_memory_mapping: [mem 0xc0000000-0xdf14dfff]
init_memory_mapping: [mem 0x00100000-0xbfffffff]
RAMDISK: [mem 0x371a0000-0x378c7fff]
ACPI: Early table checksum verification disabled
ACPI: RSDP 0x00000000000F2590 000014 (v00 CORE  )
ACPI: RSDT 0x00000000DF162424 000044 (v01 CORE   COREBOOT 00000000 CORE 00000000)
ACPI: FACP 0x00000000DF164CE8 0000F4 (v04 CORE   COREBOOT 00000000 CORE 00000000)
ACPI: DSDT 0x00000000DF1624C8 0027DA (v02 PCENG  COREBOOT 00010001 INTL 20121114)
ACPI: FACS 0x00000000DF164CA8 000040
ACPI: SPCR 0x00000000DF164DDC 000050 (v01 CORE   COREBOOT 00000000 CORE 00000000)
ACPI: HPET 0x00000000DF164E30 000038 (v01 CORE   COREBOOT 00000000 CORE 00000000)
ACPI: APIC 0x00000000DF164E68 00005C (v01 CORE   COREBOOT 00000000 CORE 00000000)
ACPI: HEST 0x00000000DF164EC8 000198 (v01 CORE   COREBOOT 00000000 CORE 00000000)
ACPI: SSDT 0x00000000DF165060 00168E (v02 AMD    ALIB     00000001 MSFT 04000000)
ACPI: SSDT 0x00000000DF1666F0 00030C (v01 AMD    POWERNOW 00000001 AMD  00000001)
ACPI: SSDT 0x00000000DF166A00 000045 (v02 CORE   COREBOOT 0000002A CORE 0000002A)
No NUMA configuration found
Faking a node at [mem 0x0000000000000000-0x000000011effffff]
NODE_DATA(0) allocated [mem 0x11effc000-0x11effffff]
Zone ranges:
  DMA      [mem 0x0000000000001000-0x0000000000ffffff]
  DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
  Normal   [mem 0x0000000100000000-0x000000011effffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000000001000-0x000000000009efff]
  node   0: [mem 0x0000000000100000-0x00000000df14dfff]
  node   0: [mem 0x0000000100000000-0x000000011effffff]
Initmem setup node 0 [mem 0x0000000000001000-0x000000011effffff]
ACPI: PM-Timer IO Port: 0x808
IOAPIC[0]: apic_id 2, version 33, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x43538210 base: 0xfed00000
smpboot: Allowing 2 CPUs, 0 hotplug CPUs
e820: [mem 0xe0000000-0xf7ffffff] available for PCI devices
Booting paravirtualized kernel on bare hardware
clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff88011ec00000 s89432 r8192 d29352 u1048576
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 1024337
Policy zone: Normal
Kernel command line: BOOT_IMAGE=/vmlinuz-4.2.5-zgsrv20080 root=/dev/mapper/aida-aida_root ro console=ttyS0,115200n8 net.ifnames=1
PID hash table entries: 4096 (order: 3, 32768 bytes)
Memory: 4014848K/4162480K available (4613K kernel code, 600K rwdata, 1656K rodata, 916K init, 452K bss, 147632K reserved, 0K cma-reserved)
Preemptible hierarchical RCU implementation.
        Build-time adjustment of leaf fanout to 64.
        RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2
NR_IRQS:4352 nr_irqs:440 16
Console: colour dummy device 80x25
console [ttyS0] enabled
clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
tsc: Fast TSC calibration using PIT
tsc: Detected 1000.033 MHz processor
Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.06 BogoMIPS (lpj=4000132)
pid_max: default: 32768 minimum: 301
ACPI: Core revision 20150619
ACPI: All ACPI Tables successfully acquired
Security Framework initialized
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
Initializing cgroup subsys blkio
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys net_prio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 6 MCE banks
Last level iTLB entries: 4KB 512, 2MB 8, 4MB 4
Last level dTLB entries: 4KB 512, 2MB 8, 4MB 4, 1GB 0
Freeing SMP alternatives memory: 20K (ffffffff8179f000 - ffffffff817a4000)
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ...
..... (found apic 0 pin 0) ...
....... failed.
...trying to set up timer as Virtual Wire IRQ...
..... works.
smpboot: CPU0: AMD G-T40E Processor (fam: 14, model: 02, stepping: 00)
Performance Events: AMD PMU driver.
... version:                0
... bit width:              48
... generic registers:      4
... value mask:             0000ffffffffffff
... max period:             00007fffffffffff
... fixed-purpose events:   0
... event mask:             000000000000000f
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
x86: Booting SMP configuration:
.... node  #0, CPUs:      #1
x86: Booted up 1 node, 2 CPUs
smpboot: Total of 2 processors activated (4000.13 BogoMIPS)
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
PGD 0
Oops: 0010 [#1] PREEMPT SMP
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.2.5-zgsrv20080 #1
Hardware name: PC Engines APU, BIOS SageBios_PCEngines_APU-45 04/05/2014
task: ffff880119ab0b00 ti: ffff880119ab4000 task.ti: ffff880119ab4000
RIP: 0010:[<0000000000000000>]  [<          (null)>]           (null)
RSP: 0018:ffff880119ab7ed0  EFLAGS: 00010246
RAX: ffff88011a005458 RBX: 0000000000000000 RCX: ffffffff816afce0
RDX: 0000000000000000 RSI: ffff88011a005458 RDI: ffff88011a005408
RBP: 0000000000000002 R08: 0000000000000000 R09: ffff880119c00008
R10: 0000000000000000 R11: 00000000000000b4 R12: ffffffff817d6e20
R13: ffff88011a005408 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff88011ec00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000000162f000 CR4: 00000000000006f0
Stack:
 ffffffff816e3d01 0000000000000001 00000000007a1308 000000000000a040
 0000000000000001 0000000000000000 ffffffff816e0a16 0000000000000001
 0000000000000000 000000000000008b ffffffff816d0fa7 ffffffff81474aa0
Call Trace:
 [<ffffffff816e3d01>] ? setup_ioapic_dest+0xcb/0xe5
 [<ffffffff816e0a16>] ? native_smp_cpus_done+0xa0/0xa8
 [<ffffffff816d0fa7>] ? kernel_init_freeable+0xd7/0x1e3
 [<ffffffff81474aa0>] ? rest_init+0x90/0x90
 [<ffffffff81474aa9>] ? kernel_init+0x9/0xe0
 [<ffffffff8147d6df>] ? ret_from_fork+0x3f/0x70
 [<ffffffff81474aa0>] ? rest_init+0x90/0x90
Code:  Bad RIP value.
RIP  [<          (null)>]           (null)
 RSP <ffff880119ab7ed0>
CR2: 0000000000000000
---[ end trace 613f6ef92efea625 ]---
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009

---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009


I haven't debugged a Linux kernel in years, how do I proceed from
here? Is there a possibility that the "bad" commit can be identified
without a bisect orgy? The APU is rather productive for my LAN and
me rebooting it many times will not please my wife ;-)

Greetings
Marc


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: PC Engines APU panics on boot since 4.2.4

[Greg KH]

On Sat, Oct 31, 2015 at 10:15:51PM +0100, Marc Haber wrote:
> Hi,
> 
> I run a PC Engines APU in my basement for local infrastructure with
> KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the
> usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors
> activated (3999.89 BogoMIPS)" message. The message I would expect
> instead of the kernel panic would be "devtmpfs: initialized", as one
> can see on a successful 4.2.3 boot with identical kernel configuration.

If you use 'git bisect' between 4.2.3 and 4.2.5, can you find the
problem commit for us to help track this down?

thanks,

greg k-h

Re: PC Engines APU panics on boot since 4.2.4

[Marc Haber]

On Sat, Oct 31, 2015 at 03:23:45PM -0700, Greg KH wrote:
> On Sat, Oct 31, 2015 at 10:15:51PM +0100, Marc Haber wrote:
> > I run a PC Engines APU in my basement for local infrastructure with
> > KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the
> > usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors
> > activated (3999.89 BogoMIPS)" message. The message I would expect
> > instead of the kernel panic would be "devtmpfs: initialized", as one
> > can see on a successful 4.2.3 boot with identical kernel configuration.
> 
> If you use 'git bisect' between 4.2.3 and 4.2.5, can you find the
> problem commit for us to help track this down?

*Sigh* sure I can, even if that means taking down the entire house
including phone for at least eight times (it's 258 commits). I'll send
my wife's complaints your way *grins*

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: PC Engines APU panics on boot since 4.2.4

[Marc Haber]

Hi,

On Sat, Oct 31, 2015 at 03:23:45PM -0700, Greg KH wrote:
> On Sat, Oct 31, 2015 at 10:15:51PM +0100, Marc Haber wrote:
> > I run a PC Engines APU in my basement for local infrastructure with
> > KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the
> > usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors
> > activated (3999.89 BogoMIPS)" message. The message I would expect
> > instead of the kernel panic would be "devtmpfs: initialized", as one
> > can see on a successful 4.2.3 boot with identical kernel configuration.
> 
> If you use 'git bisect' between 4.2.3 and 4.2.5, can you find the
> problem commit for us to help track this down?

The winner is ...
[21f751a634f3b5eaf916f940e47f327fbc6e9727] x86/ioapic: Force affinity setting in setup_ioapic_dest()
		   
After rolling this back in 4.2.5, the APU now runs fine.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: PC Engines APU panics on boot since 4.2.4

[Greg KH]

On Sun, Nov 01, 2015 at 07:26:28PM +0100, Marc Haber wrote:
> Hi,
> 
> On Sat, Oct 31, 2015 at 03:23:45PM -0700, Greg KH wrote:
> > On Sat, Oct 31, 2015 at 10:15:51PM +0100, Marc Haber wrote:
> > > I run a PC Engines APU in my basement for local infrastructure with
> > > KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the
> > > usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors
> > > activated (3999.89 BogoMIPS)" message. The message I would expect
> > > instead of the kernel panic would be "devtmpfs: initialized", as one
> > > can see on a successful 4.2.3 boot with identical kernel configuration.
> > 
> > If you use 'git bisect' between 4.2.3 and 4.2.5, can you find the
> > problem commit for us to help track this down?
> 
> The winner is ...
> [21f751a634f3b5eaf916f940e47f327fbc6e9727] x86/ioapic: Force affinity setting in setup_ioapic_dest()
> 		   
> After rolling this back in 4.2.5, the APU now runs fine.

Ok, this matches up with commit 21f751a634f3b5eaf916f940e47f327fbc6e9727
in Linus's tree.

Thomas and Mika, any thoughts about this?

Marc, do you have this same problem with Linus's tree at the moment as
well?

thanks,

greg k-h

Re: PC Engines APU panics on boot since 4.2.4

[Thomas Gleixner]

On Sun, 1 Nov 2015, Greg KH wrote:
> On Sun, Nov 01, 2015 at 07:26:28PM +0100, Marc Haber wrote:
> > Hi,
> > 
> > On Sat, Oct 31, 2015 at 03:23:45PM -0700, Greg KH wrote:
> > > On Sat, Oct 31, 2015 at 10:15:51PM +0100, Marc Haber wrote:
> > > > I run a PC Engines APU in my basement for local infrastructure with
> > > > KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the
> > > > usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors
> > > > activated (3999.89 BogoMIPS)" message. The message I would expect
> > > > instead of the kernel panic would be "devtmpfs: initialized", as one
> > > > can see on a successful 4.2.3 boot with identical kernel configuration.
> > > 
> > > If you use 'git bisect' between 4.2.3 and 4.2.5, can you find the
> > > problem commit for us to help track this down?
> > 
> > The winner is ...
> > [21f751a634f3b5eaf916f940e47f327fbc6e9727] x86/ioapic: Force affinity setting in setup_ioapic_dest()
> > 		   
> > After rolling this back in 4.2.5, the APU now runs fine.
> 
> Ok, this matches up with commit 21f751a634f3b5eaf916f940e47f327fbc6e9727
> in Linus's tree.
> 
> Thomas and Mika, any thoughts about this?
> 
> Marc, do you have this same problem with Linus's tree at the moment as
> well?

I just sent a pull request to Linus with a fix for this. Patch below.

Thanks,

	tglx

---
commit ababae44108b0e94b58eef6cb5bd830bd040a47f
Author: Werner Pawlitschko <werner.pawlitschko@arcor.de>
Date:   Tue Oct 27 09:08:04 2015 +0900

    x86/ioapic: Prevent NULL pointer dereference in setup_ioapic_dest()
    
    Commit 4857c91f0d19 changed the way how irq affinity is setup in
    setup_ioapic_dest() from using the core helper function to
    unconditionally calling the irq_set_affinity() callback of the
    underlying irq chip.
    
    That results in a NULL pointer dereference for the rare case where the
    underlying irq chip is lapic_chip which has no irq_set_affinity()
    callback. lapic_chip is occasionally used for the timer interrupt (irq
    0).
    
    The fix is simple: Check the availability of the callback instead of
    calling it unconditionally.
    
    Fixes: 4857c91f0d19 "x86/ioapic: Force affinity setting in setup_ioapic_dest()"
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    Cc: stable@vger.kernel.org

diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index bb6bfc01cb82..4f2821527014 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2547,7 +2547,9 @@ void __init setup_ioapic_dest(void)
 			mask = apic->target_cpus();
 
 		chip = irq_data_get_irq_chip(idata);
-		chip->irq_set_affinity(idata, mask, false);
+		/* Might be lapic_chip for irq 0 */
+		if (chip->irq_set_affinity)
+			chip->irq_set_affinity(idata, mask, false);
 	}
 }
 #endif

Re: PC Engines APU panics on boot since 4.2.4

[Marc Haber]

On Sun, Nov 01, 2015 at 10:34:18AM -0800, Greg KH wrote:
> Marc, do you have this same problem with Linus's tree at the moment as
> well?

I can confirm that Linus' 4.3-rc7 fails the same way, yes.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: PC Engines APU panics on boot since 4.2.4

[Marc Haber]

On Sun, Nov 01, 2015 at 07:57:47PM +0100, Thomas Gleixner wrote:
> I just sent a pull request to Linus with a fix for this. Patch below.

I can confirm that this patch, applied to the original stable 4.2.5,
works fine on the APU as well.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: PC Engines APU panics on boot since 4.2.4

[Marc Haber]

On Sun, Nov 01, 2015 at 07:57:47PM +0100, Thomas Gleixner wrote:
> I just sent a pull request to Linus with a fix for this. Patch below.

Glad to see this has made it to the 4.3 release. Thanks for the
lightning fast reaction, I really appreciate that.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421