From mboxrd@z Thu Jan 1 00:00:00 1970 From: Borislav Petkov Subject: Re: [PATCH][RT] x86: Fix an RT MCE crash Date: Thu, 30 Jun 2016 20:22:57 +0200 Message-ID: <20160630182257.GD3932@pd.tnic> References: <20160630094301.22d32ec1@gandalf.local.home> <5775316F.2020102@acm.org> <20160630115101.6337c395@gandalf.local.home> <20160630160128.GA4365@pd.tnic> <3908561D78D1C84285E8C5FCA982C28F3A14CDB9@ORSMSX114.amr.corp.intel.com> <57754B71.2000108@acm.org> <20160630170134.GA3932@pd.tnic> <57755449.7070302@acm.org> <20160630172611.GC3932@pd.tnic> <57755CC6.60506@acm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Cc: "Luck, Tony" , Steven Rostedt , "linux-rt-users@vger.kernel.org" , Corey Minyard To: Corey Minyard Return-path: Received: from mail.skyhub.de ([78.46.96.112]:41168 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751547AbcF3ScF (ORCPT ); Thu, 30 Jun 2016 14:32:05 -0400 Content-Disposition: inline In-Reply-To: <57755CC6.60506@acm.org> Sender: linux-rt-users-owner@vger.kernel.org List-ID: On Thu, Jun 30, 2016 at 12:54:14PM -0500, Corey Minyard wrote: > It won't crash. If you disable PREEMPT_RT on the 3.10-rt kernel it won't > crash (which I have tested). With PREEMPT_RT, the kernel creates a > separate thread that is woken on mce notifications. The trouble is > that the interrupts are initialized before the thread is created. Hmmm. Ok, so I don't have any idea what RT does but from looking at your splat: [ 0.164153] Call Trace: [ 0.164165] [ 0.164185] [] try_to_wake_up+0x28/0x320 [ 0.164188] [] wake_up_process+0x10/0x20 [ 0.164207] [] mce_notify_irq+0x28/0x30 [ 0.164210] [] intel_threshold_interrupt+0xb5/0xd0 [ 0.164213] [] smp_threshold_interrupt+0x1c/0x40 [ 0.164221] [] threshold_interrupt+0x6a/0x70 [ 0.164223] [ 0.164226] [] ? cmci_recheck+0x67/0x70 [ 0.164241] [] setup_local_APIC+0x276/0x283 [ 0.164259] [] native_smp_prepare_cpus+0x379/0x43b [ 0.164266] [] kernel_init_freeable+0xd7/0x21a [ 0.164270] [] ? rest_init+0x90/0x90 [ 0.164272] [] kernel_init+0x9/0x180 [ 0.164275] [] ret_from_fork+0x58/0x90 [ 0.164277] [] ? rest_init+0x90/0x90 [ 0.164295] Code: e7 ff ff 48 8b 7d 08 e8 02 1a 95 ff 5d c3 55 48 89 e5 41 54 53 48 89 fb 9c 41 5c fa bf 01 00 00 00 e8 a8 38 00 00 ba 00 01 00 00 66 0f c1 13 0f b6 ce 38 d1 74 10 0f 1f 80 00 00 00 00 f3 90 [ 0.164298] RIP [] _raw_spin_lock_irqsave+0x1d/0x50 [ 0.164298] RSP [ 0.164299] CR2: 0000000000000600 [ 0.656225] ---[ end trace 0000000000000001 ]--- [ 0.656233] Kernel panic - not syncing: Fatal exception in interrupt we're 0.16 seconds within the boot and we're just initializing the local APIC and the moment that happens, we get a thresholding APIC interrupt. So how can interrupts be initialized before that? I'm genuinely asking because I can't imagine how CMCI can get initialized *after* the local APIC init. Because, we do init CMCI in identify_cpu()->mcheck_cpu_init() and that happens earlier than your splat. You can even see where it happens in dmesg: [ 0.049270] mce: CPU supports 22 MCE banks [ 0.049383] CPU0: Thermal monitoring enabled (TM1) First line is __mcheck_cpu_cap_init(), second is intel_init_thermal(). The CMCI initialization is done right after it in void mce_intel_feature_init(struct cpuinfo_x86 *c) { intel_init_thermal(c); intel_init_cmci(); but wait!, this is the upstream kernel. Where can I look at 3.10-rt sources? -- Regards/Gruss, Boris. ECO tip #101: Trim your mails when you reply.