From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from [59.151.112.132] (helo=heian.cn.fujitsu.com) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bMYUL-0006RG-Ac for kexec@lists.infradead.org; Mon, 11 Jul 2016 10:30:59 +0000 From: "Wei, Jiangang" Subject: Re: [PATCH v2] kexec: Fix kdump failure with notsc Date: Mon, 11 Jul 2016 10:30:23 +0000 Message-ID: <1468232891.15074.45.camel@localhost> References: <1467886646-2638-1-git-send-email-weijg.fnst@cn.fujitsu.com> <87r3b5z626.fsf@x220.int.ebiederm.org> <20160708073828.GA3216@gmail.com> In-Reply-To: <20160708073828.GA3216@gmail.com> Content-Language: en-US Content-ID: <2CF79CEF14AEFA4890B058F1FA51D9C3@fujitsu.local> MIME-Version: 1.0 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "kexec" Errors-To: kexec-bounces+dwmw2=infradead.org@lists.infradead.org To: "mingo@kernel.org" Cc: "fenghua.yu@intel.com" , "x86@kernel.org" , "kexec@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "mingo@redhat.com" , "ebiederm@xmission.com" , "hpa@zytor.com" , "tglx@linutronix.de" SGkgLCBJbmdvDQoNCk9uIEZyaSwgMjAxNi0wNy0wOCBhdCAwOTozOCArMDIwMCwgSW5nbyBNb2xu YXIgd3JvdGU6DQo+ICogRXJpYyBXLiBCaWVkZXJtYW4gPGViaWVkZXJtQHhtaXNzaW9uLmNvbT4g d3JvdGU6DQo+IA0KPiA+IFNpZ2guICBDYW4gd2UgcGxlYXNlIGp1c3QgZG8gdGhlIHdvcmsgdG8g cmlwIG91dCB0aGUgYXBpYyBzaHV0ZG93biBjb2RlIGZyb20gdGhlIA0KPiA+IGtleGVjIG9uIHBh bmljIGNvZGUgcGF0aD8NCj4gPiANCj4gPiBJIGZvcmdldHRpbmcgZGV0YWlscyBidXQgdGhlIG9u bHkgcmVhc29uIHdlIGhhdmUgZG8gYW55IGFwaWMgc2h1dGRvd24gaXMgYnVncyBpbiANCj4gPiBv bGRlciBrZXJuZWxzIHRoYXQgY291bGQgbm90IGluaXRpYWxpemUgYSBzeXN0ZW0gcHJvcGVybHkg aWYgd2UgZGlkIG5vdCBzaHV0IA0KPiA+IGRvd24gdGhlIGFwaWNzLg0KPiA+IA0KPiA+IEkgY2Vy dGFpbmx5IGRvbid0IHNlZSBhbiBpc3N1ZSB3aXRoIGdvb2Z5IGNhc2VzIGxpa2Ugbm90c2Mgbm90 IHdvcmtpbmcgb24gYSANCj4gPiBjcmFzaCBjYXB0dXJlIGtlcm5lbCBpZiB3ZSBhcmUgbm90IGlu aXRpYWxpemluZyB0aGUgaGFyZHdhcmUgcHJvcGVybHkuDQo+ID4gDQo+ID4gVGhlIHN0cmF0ZWd5 IHJlYWxseSBuZWVkcyB0byBiZSB0byBvbmx5IGRvIHRoZSBhYnNvbHV0ZWx5IGVzc2VudGlhbCBo YXJkd2FyZSANCj4gPiBzaHV0ZG93biBpbiB0aGUgY3Jhc2hpbmcga2VybmVsLCBldmVyeSBhZGlu dGlvbmFsIGxpbmUgb2YgY29kZSB3ZSBleGVjdXRlIGluIHRoZSANCj4gPiBjcmFzaGluZyBrZXJu ZWwgaW5jcmVhc2VzIG91ciBjaGFuY2VzIG9mIGhpdHRpbmcgYSBidWcuDQo+IA0KPiBGdWxseSBh Z3JlZWQuDQo+IA0KPiA+IFVuZGVyIHRoYXQgcG9saWN5IHRoaW5ncyBsaWtlIHJlcXVyaW5nIHdl IGRvbid0IHBhc3MgYm9vdCBvcHRpb25zIHRoYXQgaW5oaWJpdCANCj4gPiB0aGUgZHVtcCBjYXRw dXJlIGtlcm5lbCBmcm9tIGluaXRpYWxpemluZyB0aGUgaGFyZHdhcmUgZnJvbSBhIHJhbmRvbSBz dGF0ZSBhcmUgDQo+ID4gcmVhc29uYWJsZSByZXF1aXJlbWVudHMuICBBS0EgSSBkb24ndCBzZWUg YW55IGp1c3RpZmljYXRpb24gaW4gdGhpcyBhcyB0byB3aHkgd2UgDQo+ID4gd291bGQgZXZlbiB3 YW50IHRvIHN1cHBvcnQgbm90c2Mgb24gdGhlIGR1bXAgY2FwdHVyZSBrZXJuZWwuICBFc3BlY2lh bGx5IHdoZW4gDQo+ID4gdGhpbmdzIGNsZWFybHkgd29yayB3aGVuIHRoYXQgb3B0aW9uIGlzIG5v dCBzcGVjaWZpZWQuDQo+IA0KPiBTbyBhdCBsZWFzdCBvbiB0aGUgc3VyZmFjZSBpdCBhcHBlYXJz ICdzdXJwcmlzaW5nJyB0aGF0IHRoZSAnbm90c2MnIG9wdGlvbiAod2hpY2gsIA0KPiBzdXBwb3Nl ZGx5LCBkaXNhYmxlcyBUU0MgaGFuZGxpbmcpIGludGVyZmVyZXMgd2l0aCBiZWluZyBhYmxlIHRv IGZ1bGx5IGJvb3QuIEV2ZW4gDQo+IGlmICdub3RzYycgaXMgc3BlY2lmaWVkIHdlIGFyZSBzdGls bCB1c2luZyB0aGUgbG9jYWwgQVBJQywgcmlnaHQ/DQoNCkluIG1vc3QgY2FzZSwgIEl0J3Mgbm8g cHJvYmxlbSB0aGF0IHVzaW5nIGxvY2FsIEFQSUMgd2hpbGUgbm90c2MgaXMNCnNwZWNpZmllZC4N CkJ1dCBub3QgZm9yIGtkdW1wLg0KDQpXZSBjYW4gZ2V0IGV2aWRlbmNlLCBFc3BlY2lhbGx5IGZy b20gIlNwdXJpb3VzIExBUElDIHRpbWVyIGludGVycnVwdCBvbg0KY3B1IDAiLg0KDQojIyNzZXJp YWwgbG9nLA0KDQpbICAgIDAuMDAwMDAwXSBOUl9JUlFTOjUyNDU0NCBucl9pcnFzOjI1NiAxNg0K WyAgICAwLjAwMDAwMF0gU3B1cmlvdXMgTEFQSUMgdGltZXIgaW50ZXJydXB0IG9uIGNwdSAwDQpb ICAgIDAuMDAwMDAwXSBDb25zb2xlOiBjb2xvdXIgZHVtbXkgZGV2aWNlIDgweDI1DQpbICAgIDAu MDAwMDAwXSBjb25zb2xlIFt0dHkwXSBlbmFibGVkDQpbICAgIDAuMDAwMDAwXSBjb25zb2xlIFt0 dHlTMF0gZW5hYmxlZA0KWyAgICAwLjAwMDAwMF0gdHNjOiBGYXN0IFRTQyBjYWxpYnJhdGlvbiB1 c2luZyBQSVQNClsgICAgMC4wMDAwMDBdIHRzYzogRGV0ZWN0ZWQgMjA5OS45NDcgTUh6IHByb2Nl c3Nvcg0KWyAgICAwLjAwMDAwMF0gQ2FsaWJyYXRpbmcgZGVsYXkgbG9vcC4uLg0KDQoNCkR1ZSB0 byB0aGUgbG9jYWwgYXBpYyBhbmQgbG9jYWwgYXBpYyB0aW1lciBoYXNuJ3QgYmVlbiBzZXR1cCBh bmQgZW5hYmxlZA0KZnVsbHksIFRoZSBldmVudF9oYW5kbGVyIG9mIGNsb2NrIGV2ZW50IGlzIE5V TEwuDQoNCiMjI2NvZGVzLA0KDQpzdGF0aWMgdm9pZCBsb2NhbF9hcGljX3RpbWVyX2ludGVycnVw dCh2b2lkKQ0Kew0KICAgIGludCBjcHUgPSBzbXBfcHJvY2Vzc29yX2lkKCk7DQogICAgc3RydWN0 IGNsb2NrX2V2ZW50X2RldmljZSAqZXZ0ID0gJnBlcl9jcHUobGFwaWNfZXZlbnRzLCBjcHUpOw0K DQogICAgLyoNCiAgICAgKiBOb3JtYWxseSB3ZSBzaG91bGQgbm90IGJlIGhlcmUgdGlsbCBMQVBJ QyBoYXMgYmVlbiBpbml0aWFsaXplZA0KYnV0DQogICAgICogaW4gc29tZSBjYXNlcyBsaWtlIGtk dW1wLCBpdHMgcG9zc2libGUgdGhhdCB0aGVyZSBpcyBhIHBlbmRpbmcNCkxBUElDDQogICAgICog dGltZXIgaW50ZXJydXB0IGZyb20gcHJldmlvdXMga2VybmVsJ3MgY29udGV4dCBhbmQgaXMgZGVs aXZlcmVkDQppbg0KICAgICAqIG5ldyBrZXJuZWwgdGhlIG1vbWVudCBpbnRlcnJ1cHRzIGFyZSBl bmFibGVkLg0KICAgICAqDQogICAgICogSW50ZXJydXB0cyBhcmUgZW5hYmxlZCBlYXJseSBhbmQg TEFQSUMgaXMgc2V0dXAgbXVjaCBsYXRlciwgaGVuY2UNCiAgICAgKiBpdHMgcG9zc2libGUgdGhh dCB3aGVuIHdlIGdldCBoZXJlIGV2dC0+ZXZlbnRfaGFuZGxlciBpcyBOVUxMLg0KICAgICAqIENo ZWNrIGZvciBldmVudF9oYW5kbGVyIGJlaW5nIE5VTEwgYW5kIGRpc2NhcmQgdGhlIGludGVycnVw dCBhcw0KICAgICAqIHNwdXJpb3VzLg0KICAgICAqLw0KICAgIGlmICghZXZ0LT5ldmVudF9oYW5k bGVyKSB7DQogICAgICAgIHByX3dhcm5pbmcoIlNwdXJpb3VzIExBUElDIHRpbWVyIGludGVycnVw dCBvbiBjcHUgJWRcbiIsIGNwdSk7DQogICAgICAgIC8qIFN3aXRjaCBpdCBvZmYgKi8NCiAgICAg ICAgbGFwaWNfdGltZXJfc2h1dGRvd24oZXZ0KTsNCiAgICAgICAgcmV0dXJuOw0KICAgIH0NCg0K ICAgICAuLi4uLi4uLi4uLi4uDQp9DQoNCg0KSU1ITywgDQpJZiB3ZSBzcGVjaWZ5IG5vdHNjLCB0 aGUgZHVtcC1jYXB0dXJlIGtlcm5lbCB3YWl0cyBmb3IgamlmZmllcyBiZWluZw0KdXBkYXRlZCBl YXJseSBhbmQgTEFQSUMgYW5kIHRpbWVyIGFyZSBzZXR1cCBtdWNoIGxhdGVyLCB3aGljaCBjYXVz ZXMgbm8NCnRpbWVyIGludGVycnVwdHMgaXMgcGFzc2VkIHRvIEJTUC4gYXMgZm9sbG93aW5nLA0K DQpzdGFydF9rZXJuZWwgIC0tPg0KMSktPiBjYWxpYnJhdGVfZGVsYXkoKSAgLT4gY2FsaWJyYXRl X2RlbGF5X2NvbnZlcmdlKCkgICMgaGFuZyBhbmQgd2FpdA0KZm9yIGppZmZpZXMgY2hhbmdlZA0K ICAgICAgICAgICAgICAgICAgIA0KMiktPiByZXN0X2luaXQoKSAtPiBrZXJuZWxfaW5pdCgpIC0+ IC4uLi4gLT4gYXBpY19ic3Bfc2V0dXAoKSAtPg0Kc2V0dXBfbG9jYWxfQVBJQygpDQoNCi0+IHNl dHVwX3BlcmNwdV9jbG9ja2V2KCkuDQoNCnRoZSBzZXR1cF9wZXJjcHVfY2xvY2tldiBwb2ludHMg c2V0dXBfYm9vdF9BUElDX2Nsb2NrKCkgd2hpY2ggdXNlZCB0bw0Kc2V0dXAgdGhlIGJvb3QgQVBJ QyBhbmQgdGltZXIuDQoNCg0KPiBTbyBpdCBtaWdodCBiZSBhIGdvb2QgaWRlYSB0byBmaW5kIHRo ZSByb290IGNhdXNlIG9mIHRoaXMgYm9vdHVwIGZyYWdpbGl0eSBldmVuIGlmIA0KPiAnbm90c2Mn IGlzIHNwZWNpZmllZC4gQW5kIEkgZnVsbHkgYWdyZWUgdGhhdCBpdCBzaG91bGQgYmUgZml4ZWQg aW4gdGhlIGJvb3R1cCBwYXRoIA0KPiBvZiB0aGUgZHVtcCBrZXJuZWwsIG5vdCB0aGUgY3Jhc2gg a2VybmVsIHJlYm9vdCBwYXRoLg0KDQpCZWNhdXNlIHRoZSBsYXBpYyBhbmQgdGltZXIgYXJlIG5v dCByZWFkeSB3aGVuIGR1bXAtY2FwdHVyZSB3YWl0cyB0aGVtDQp0byB1cGRhdGUgdGhlIGppZmZp ZXMgdmFsdWUuIHNvIEkgc3VnZ2VzdCB0byBwdXQgQVBJQyBpbiBsZWdhY3kgbW9kZSBpbg0KbG9j YWxfYXBpY190aW1lcl9pbnRlcnJ1cHQoKSB0ZW1wb3JhcmlseSwgd2hpY2ggaW4gdGhlIGJvb3R1 cCBwYXRoIG9mDQpkdW1wIGtlcm5lbC4gDQoNCmRpZmYgLS1naXQgYS9hcmNoL3g4Ni9rZXJuZWwv YXBpYy9hcGljLmMgYi9hcmNoL3g4Ni9rZXJuZWwvYXBpYy9hcGljLmMNCmluZGV4IGRjYjUyODUw YTI4Zi4uYWYzYmU5Mzk5N2VkIDEwMDY0NA0KLS0tIGEvYXJjaC94ODYva2VybmVsL2FwaWMvYXBp Yy5jDQorKysgYi9hcmNoL3g4Ni9rZXJuZWwvYXBpYy9hcGljLmMNCkBAIC04NzksNiArODc5LDcg QEAgc3RhdGljIHZvaWQgbG9jYWxfYXBpY190aW1lcl9pbnRlcnJ1cHQodm9pZCkNCiAgICAgICAg ICAgICAgICBwcl93YXJuaW5nKCJTcHVyaW91cyBMQVBJQyB0aW1lciBpbnRlcnJ1cHQgb24gY3B1 ICVkXG4iLA0KY3B1KTsNCiAgICAgICAgICAgICAgICAvKiBTd2l0Y2ggaXQgb2ZmICovDQogICAg ICAgICAgICAgICAgbGFwaWNfdGltZXJfc2h1dGRvd24oZXZ0KTsNCisgICAgICAgICAgICAgZGlz YWJsZV9JT19BUElDKCk7DQogICAgICAgICAgICAgICAgcmV0dXJuOw0KICAgICAgICB9DQoNCkFu ZCB0aGUgbmV3IHNvbHV0aW9uIGNhbiBmaXggdGhlIHByb2JsZW0uDQpXaGF04oCYcyB5b3VyIG9w aW5pb24gYWJvdXQgaXQ/DQoNClRoYW5rcywNCndlaQ0KDQo+IA0KPiBUaGFua3MsDQo+IA0KPiAJ SW5nbw0KPiANCj4gDQoNCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18Ka2V4ZWMgbWFpbGluZyBsaXN0CmtleGVjQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0 cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9rZXhlYwo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758335AbcGKKae (ORCPT ); Mon, 11 Jul 2016 06:30:34 -0400 Received: from cn.fujitsu.com ([59.151.112.132]:33454 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1753215AbcGKKac (ORCPT ); Mon, 11 Jul 2016 06:30:32 -0400 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="8585812" From: "Wei, Jiangang" To: "mingo@kernel.org" CC: "kexec@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "fenghua.yu@intel.com" , "ebiederm@xmission.com" , "x86@kernel.org" , "hpa@zytor.com" , "mingo@redhat.com" Subject: Re: [PATCH v2] kexec: Fix kdump failure with notsc Thread-Topic: [PATCH v2] kexec: Fix kdump failure with notsc Thread-Index: AQHR2DkmJRcl93aqHkuGcdrihkYm5qANRODQgABcAACABOZqgA== Date: Mon, 11 Jul 2016 10:30:23 +0000 Message-ID: <1468232891.15074.45.camel@localhost> References: <1467886646-2638-1-git-send-email-weijg.fnst@cn.fujitsu.com> <87r3b5z626.fsf@x220.int.ebiederm.org> <20160708073828.GA3216@gmail.com> In-Reply-To: <20160708073828.GA3216@gmail.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.167.226.50] Content-Type: text/plain; charset="utf-8" Content-ID: <2CF79CEF14AEFA4890B058F1FA51D9C3@fujitsu.local> MIME-Version: 1.0 X-yoursite-MailScanner-ID: 15745498F58C.AB561 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: weijg.fnst@cn.fujitsu.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id u6BAUcKa017310 Hi , Ingo On Fri, 2016-07-08 at 09:38 +0200, Ingo Molnar wrote: > * Eric W. Biederman wrote: > > > Sigh. Can we please just do the work to rip out the apic shutdown code from the > > kexec on panic code path? > > > > I forgetting details but the only reason we have do any apic shutdown is bugs in > > older kernels that could not initialize a system properly if we did not shut > > down the apics. > > > > I certainly don't see an issue with goofy cases like notsc not working on a > > crash capture kernel if we are not initializing the hardware properly. > > > > The strategy really needs to be to only do the absolutely essential hardware > > shutdown in the crashing kernel, every adintional line of code we execute in the > > crashing kernel increases our chances of hitting a bug. > > Fully agreed. > > > Under that policy things like requring we don't pass boot options that inhibit > > the dump catpure kernel from initializing the hardware from a random state are > > reasonable requirements. AKA I don't see any justification in this as to why we > > would even want to support notsc on the dump capture kernel. Especially when > > things clearly work when that option is not specified. > > So at least on the surface it appears 'surprising' that the 'notsc' option (which, > supposedly, disables TSC handling) interferes with being able to fully boot. Even > if 'notsc' is specified we are still using the local APIC, right? In most case, It's no problem that using local APIC while notsc is specified. But not for kdump. We can get evidence, Especially from "Spurious LAPIC timer interrupt on cpu 0". ###serial log, [ 0.000000] NR_IRQS:524544 nr_irqs:256 16 [ 0.000000] Spurious LAPIC timer interrupt on cpu 0 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2099.947 MHz processor [ 0.000000] Calibrating delay loop... Due to the local apic and local apic timer hasn't been setup and enabled fully, The event_handler of clock event is NULL. ###codes, static void local_apic_timer_interrupt(void) { int cpu = smp_processor_id(); struct clock_event_device *evt = &per_cpu(lapic_events, cpu); /* * Normally we should not be here till LAPIC has been initialized but * in some cases like kdump, its possible that there is a pending LAPIC * timer interrupt from previous kernel's context and is delivered in * new kernel the moment interrupts are enabled. * * Interrupts are enabled early and LAPIC is setup much later, hence * its possible that when we get here evt->event_handler is NULL. * Check for event_handler being NULL and discard the interrupt as * spurious. */ if (!evt->event_handler) { pr_warning("Spurious LAPIC timer interrupt on cpu %d\n", cpu); /* Switch it off */ lapic_timer_shutdown(evt); return; } ............. } IMHO, If we specify notsc, the dump-capture kernel waits for jiffies being updated early and LAPIC and timer are setup much later, which causes no timer interrupts is passed to BSP. as following, start_kernel --> 1)-> calibrate_delay() -> calibrate_delay_converge() # hang and wait for jiffies changed 2)-> rest_init() -> kernel_init() -> .... -> apic_bsp_setup() -> setup_local_APIC() -> setup_percpu_clockev(). the setup_percpu_clockev points setup_boot_APIC_clock() which used to setup the boot APIC and timer. > So it might be a good idea to find the root cause of this bootup fragility even if > 'notsc' is specified. And I fully agree that it should be fixed in the bootup path > of the dump kernel, not the crash kernel reboot path. Because the lapic and timer are not ready when dump-capture waits them to update the jiffies value. so I suggest to put APIC in legacy mode in local_apic_timer_interrupt() temporarily, which in the bootup path of dump kernel. diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index dcb52850a28f..af3be93997ed 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -879,6 +879,7 @@ static void local_apic_timer_interrupt(void) pr_warning("Spurious LAPIC timer interrupt on cpu %d\n", cpu); /* Switch it off */ lapic_timer_shutdown(evt); + disable_IO_APIC(); return; } And the new solution can fix the problem. What‘s your opinion about it? Thanks, wei > > Thanks, > > Ingo > >