From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-we0-x229.google.com ([2a00:1450:400c:c03::229]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1Ya2l6-0003ek-ON for kexec@lists.infradead.org; Mon, 23 Mar 2015 13:51:14 +0000 Received: by webck51 with SMTP id ck51so11565722web.2 for ; Mon, 23 Mar 2015 06:50:50 -0700 (PDT) Date: Mon, 23 Mar 2015 14:50:46 +0100 From: Ingo Molnar Subject: Re: [PATCH v2] kernel/panic/kexec: fix "crash_kexec_post_notifiers" option issue in oops path Message-ID: <20150323135046.GA25012@gmail.com> References: <54F9D645.2050008@jp.fujitsu.com> <20150323034752.GD2068@dhcp-16-105.nay.redhat.com> <20150323071943.GA22765@gmail.com> <20150323133710.GA3172@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20150323133710.GA3172@redhat.com> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "kexec" Errors-To: kexec-bounces+dwmw2=infradead.org@lists.infradead.org To: Vivek Goyal Cc: Baoquan He , kexec@lists.infradead.org, linux-kernel@vger.kernel.org, hidehiro.kawai.ez@hitachi.com, =?utf-8?B?IkhhdGF5YW1hLCBEYWlzdWtlL+eVkeWxsSDlpKfovJQi?= , mingo@redhat.com, ebiederm@xmission.com, masami.hiramatsu.pt@hitachi.com, akpm@linux-foundation.org, bp@suse.de CiogVml2ZWsgR295YWwgPHZnb3lhbEByZWRoYXQuY29tPiB3cm90ZToKCj4gT24gTW9uLCBNYXIg MjMsIDIwMTUgYXQgMDg6MTk6NDNBTSArMDEwMCwgSW5nbyBNb2xuYXIgd3JvdGU6Cj4gPiAKPiA+ ICogQmFvcXVhbiBIZSA8YmhlQHJlZGhhdC5jb20+IHdyb3RlOgo+ID4gCj4gPiA+IENDIG1vcmUg cGVvcGxlIC4uLgo+ID4gPiAKPiA+ID4gT24gMDMvMDcvMTUgYXQgMDE6MzFhbSwgIkhhdGF5YW1h LCBEYWlzdWtlL+eVkeWxsSDlpKfovJQiIHdyb3RlOgo+ID4gPiA+IFRoZSBjb21taXQgZjA2ZTUx NTNmNGFlMmUyZjNiMDMwMGYwZTI2MGU0MGNiN2ZlZmQ0NSBpbnRyb2R1Y2VkCj4gPiA+ID4gImNy YXNoX2tleGVjX3Bvc3Rfbm90aWZpZXJzIiBrZXJuZWwgYm9vdCBvcHRpb24sIHdoaWNoIHRvZ2ds ZXMKPiA+ID4gPiB3aGVhdGhlciBwYW5pYygpIGNhbGxzIGNyYXNoX2tleGVjKCkgYmVmb3JlIHBh bmljX25vdGlmaWVycyBhbmQgZHVtcAo+ID4gPiA+IGttc2cgb3IgYWZ0ZXIuCj4gPiA+ID4gCj4g PiA+ID4gVGhlIHByb2JsZW0gaXMgdGhhdCB0aGUgY29tbWl0IG92ZXJsb29rcyBwYW5pY19vbl9v b3BzIGtlcm5lbCBib290Cj4gPiA+ID4gb3B0aW9uLiBJZiBpdCBpcyBlbmFibGVkLCBjcmFzaF9r ZXhlYygpIGlzIGNhbGxlZCBkaXJlY3RseSB3aXRob3V0Cj4gPiA+ID4gZ29pbmcgdGhyb3VnaCBw YW5pYygpIGluIG9vcHMgcGF0aC4KPiA+ID4gPiAKPiA+ID4gPiBUbyBmaXggdGhpcyBpc3N1ZSwg dGhpcyBwYXRjaCBhZGRzIGEgY2hlY2sgdG8KPiA+ID4gPiAiY3Jhc2hfa2V4ZWNfcG9zdF9ub3Rp ZmllcnMiIGluIHRoZSBjb25kaXRpb24gb2Yga2V4ZWNfc2hvdWxkX2NyYXNoKCkuCj4gPiA+ID4g Cj4gPiA+ID4gQWxzbywgcHV0IGEgY29tbWVudCBpbiBrZXhlY19zaG91bGRfY3Jhc2goKSB0byBl eHBsYWluIG5vdCBvYnZpb3VzCj4gPiA+ID4gdGhpbmdzIG9uIHRoaXMgcGF0Y2guCj4gPiA+ID4g Cj4gPiA+ID4gU2lnbmVkLW9mZi1ieTogSEFUQVlBTUEgRGFpc3VrZSA8ZC5oYXRheWFtYUBqcC5m dWppdHN1LmNvbT4KPiA+ID4gPiBBY2tlZC1ieTogQmFvcXVhbiBIZSA8YmhlQHJlZGhhdC5jb20+ Cj4gPiA+ID4gVGVzdGVkLWJ5OiBIaWRlaGlybyBLYXdhaSA8aGlkZWhpcm8ua2F3YWkuZXpAaGl0 YWNoaS5jb20+Cj4gPiA+ID4gUmV2aWV3ZWQtYnk6IE1hc2FtaSBIaXJhbWF0c3UgPG1hc2FtaS5o aXJhbWF0c3UucHRAaGl0YWNoaS5jb20+Cj4gPiA+ID4gLS0tCj4gPiA+ID4gIGluY2x1ZGUvbGlu dXgva2VybmVsLmggfCAgMyArKysKPiA+ID4gPiAga2VybmVsL2tleGVjLmMgICAgICAgICB8IDEx ICsrKysrKysrKysrCj4gPiA+ID4gIGtlcm5lbC9wYW5pYy5jICAgICAgICAgfCAgMiArLQo+ID4g PiA+ICAzIGZpbGVzIGNoYW5nZWQsIDE1IGluc2VydGlvbnMoKyksIDEgZGVsZXRpb24oLSkKPiA+ IAo+ID4gVGhpcyBpcyBoYWNrIHVwb24gaGFjaywgYnV0IHdoeSB3YXMgdGhpcyBjcmFwIG1lcmdl ZCBpbiB0aGUgZmlyc3QgCj4gPiBwbGFjZT8KPiA+IAo+ID4gSSBzZWUgdHdvIHByb2JsZW1zIGp1 c3QgYnkgY3Vyc29yeSByZXZpZXc6Cj4gPiAKPiA+IDEpCj4gPiAKPiA+IEZpcnN0bHksIHRoZSBy ZWFsIGJ1ZyBpbjoKPiA+IAo+ID4gICBmMDZlNTE1M2Y0YWUgKCJrZXJuZWwvcGFuaWMuYzogYWRk ICJjcmFzaF9rZXhlY19wb3N0X25vdGlmaWVycyIgb3B0aW9uIGZvciBrZHVtcCBhZnRlciBwYW5p Y19ub3RpZmVycyIpCj4gPiAKPiA+IFdhcyB0aGF0IGNyYXNoX2tleGVjKCkgd2FzIGNhbGxlZCB1 bmNvbmRpdGlvbmFsbHkgYWZ0ZXIgbm90aWZpZXJzIHdlcmUgCj4gPiBjYWxsZWQsIHdoaWNoIHNo b3VsZCBiZSBmaXhlZCB2aWEgdGhlIHNpbXBsZSBwYXRjaCBiZWxvdyAodW50ZXN0ZWQpLiAKPiA+ IExvb2tzIG11Y2ggc2ltcGxlciB0aGFuIHlvdXIgZml4Lgo+ID4gCj4gCj4gSGkgSW5nbywKPiAK PiBBZ3JlZWQuIFlvdXIgcGF0Y2ggbG9va3MgZ29vZC4KCkluIGNhc2UgeW91IHdhbnQgdGhhdCBz aW1wbGVyIGZpeCBhbmQgbmVlZCBteSBTT0I6CgogIFNpZ25lZC1vZmYtYnk6IEluZ28gTW9sbmFy IDxtaW5nb0BrZXJuZWwub3JnPgoKKGJ1dCBJIGhhdmUgbm90IHRlc3RlZCBpdC4pCgo+ID4gU2Vj b25kbHksIGFuZCBtb3JlIGltcG9ydGFudGx5LCB0aGUgd2hvbGUgcHJlbWlzZSBvZiBjb21taXQg Cj4gPiBmMDZlNTE1M2Y0YWUgaXMgYnJva2VuIElNSE86Cj4gPiAKPiA+ICAiVGhpcyBjYW4gaGVs cCByYXJlIHNpdHVhdGlvbnMgd2hlcmUga2R1bXAgZmFpbHMgYmVjYXVzZSBvZiB1bnN0YWJsZQo+ ID4gICBjcmFzaGVkIGtlcm5lbCBvciBoYXJkd2FyZSBmYWlsdXJlIChtZW1vcnkgY29ycnVwdGlv biBvbiBjcml0aWNhbAo+ID4gICBkYXRhL2NvZGUpIgo+ID4gCj4gPiB3dGY/Cj4gPiAKPiA+IElm IHRoZSBrZXJuZWwgY3Jhc2hlZCBkdWUgdG8gYSBrZXJuZWwgY3Jhc2gsIHRoZW4gdGhlIGtlcm5l bCBib290aW5nIAo+ID4gdXAgaW4gd2hhdGV2ZXIgaGFyZHdhcmUgc3RhdGUgc2hvdWxkIGJlIGFi bGUgdG8gZG8gYSBjbGVhbiBib290dXAuIFRoZSAKPiA+IGZpeCBmb3IgdGhvc2UgJ3JhcmUgc2l0 dWF0aW9ucycgc2hvdWxkIGJlIHRvIGZpeCB0aGUgcmVhbCBidWcgKGZvciAKPiA+IGV4YW1wbGUg YnkgbWFraW5nIGhhcmR3YXJlIGRyaXZlciBpbml0IChvciBkZWluaXQpIHNlcXVlbmNlcyBtb3Jl IAo+ID4gcm9idXN0KSwgbm90IHRvIHBhcGVyIGl0IG92ZXIgYnkgb3JkZXJpbmcgYXJvdW5kIGNy YXNoLXRpbWUgc2VxdWVuY2VzIAo+ID4gLi4uCj4gPiAKPiA+IElmIGl0IGNyYXNoZWQgZHVlIHRv IHNvbWUgaGFyZHdhcmUgZmFpbHVyZSwgdGhlcmUncyBsaXRlcmFsbHkgYW4gCj4gPiBpbmZpbml0 ZSBhbW91bnQgb2YgZmFpbHVyZSBtb2RlcyB0aGF0IG1heSBvciBtYXkgbm90IGJlIGltcGFjdGVk IGJ5IAo+ID4ga2V4ZWMgY3Jhc2gtdGltZSBoYW5kbGluZyBvcmRlcmluZy4gV2UgZG9uJ3Qgd2Fu dCB0byBwdXQgYSB6aWxsaW9uIAo+ID4gc3VjaCBmbGFncyBpbnRvIHRoZSBrZXJuZWwgcHJvcGVy IGp1c3QgdG8gYWxsb3cgdGhlIHBlcnR1cmJhdGlvbiBvZiAKPiA+IHRoZSBrZXJuZWwuCj4gCj4g SSB0aGluayBvbmUgb2YgdGhlIG1vdGl2YXRpb25zIGJlaGluZCB0aGlzIHBhdGNoIHdhcyBjYWxs IHRvIGttc2dfZHVtcCgpLgo+IFNvbWUgdmVuZG9ycyBoYXZlIGJlZW4gd2FudGluZyB0byBoYXZl IHRoZSBjYXBhYmlsaXR5IHRvIHNhdmUga2VybmVsIGxvZ3MKPiB0byBzb21lIE5WUkFNIGJlZm9y ZSB0cmFuc2l0aW9uIHRvIHNlY29uZCBrZXJuZWwgaGFwcGVucy4gVGhlaXIgYXJndW1lbnQKPiBp cyB0aGF0IGtkdW1wIGRvZXMgbm90IHN1Y2NlZWQgYWxsIHRoZSB0aW1lIGFuZCBpZiBrZHVtcCBk b2VzIG5vdCBzdWNjZWVkCj4gdGhlbiBhdGxlYXN0IHRoZXkgaGF2ZSBzb21ldGhpbmcgdG8gd29y ayB3aXRoIChrZXJuZWwgbG9ncyByZXRyaWV2ZWQKPiBmcm9tIHBzdG9yZSBpbnRlcmZhY2UpLgoK RG9lc24ndCBwc3RvcmUgYXR0YWNoIGl0c2VsZiB0byBwcmludGsgaXRzZWxmPyBBRkFJQ1MgaXQg ZG9lczoKCiBmcy9wc3RvcmUvcGxhdGZvcm0uYzogICByZWdpc3Rlcl9jb25zb2xlKCZwc3RvcmVf Y29uc29sZSk7CgpzbyB0aGUgcHJpbnRrIGxvZyBsZWFkaW5nIHVwIHRvIGFuZCBpbmNsdWRpbmcg dGhlIGNyYXNoIHNob3VsZCBiZSAKYXZhaWxhYmxlLCByZWdhcmRsZXNzIG9mIHRoaXMgcGF0Y2gu IFdoYXQgYW0gSSBtaXNzaW5nPwoKPiBOb3QgdGhhdCBJIGFncmVlIGZ1bGx5IHdpdGggdGhpcyBh cyBwcm9ibGVtIG1pZ2h0IGhhcHBlbiB3aGlsZSB3ZSAKPiB0cnkgdG8gcnVuIHBhbmljX25vdGlm aWVycyBvciBrbXNnX2R1bXAgaG9va3MgYW5kIG5ldmVyIHRyYW5zaXRpb24gCj4gaW50byBrZHVt cCBrZXJuZWwuCgpidHcuLCB0aGlzIGlzIHRoZSBiaWcgcHJvYmxlbSB3aXRoICdub3RpZmllcnMn IGluIGdlbmVyYWw6IHRoZXkgYXJlIApvcGFxdWUgd2l0aCBiYXJlbHkgYW55IHNlbWFudGljcyBk ZWZpbmVkLCBhbmQgYSBzb3VyY2Ugb2YgY29uc3RhbnQgCmNvbmZ1c2lvbi4KCj4gQW5kIGl0IGhh cyBiZWVuIGxpdGVyYWxseSB5ZWFycyBzaW5jZSBzb21lIGRldmVsb3BlcnMgaGF2ZSBiZWVuIAo+ IHB1c2hpbmcgZm9yIGFsbG93aW5nIHRvIHJ1biBwYW5pYyBub3RpZmllcnMgYmVmb3JlIGNyYXNo X2tleGVjKCkuIAo+IEVyaWMgQmllZGVybWFuIGhhcyBiZWVuIHB1c2hpbmcgYmFjayBzYXlpbmcg aXQgcmVkdWNlcyB0aGUgCj4gcmVsaWFiaWxpdHkgb2Yga2R1bXAgb3BlcmF0aW9uIHNvIHRoaXMg aXMgbm90IGFjY2VwdGFibGUuCgpTbyB3aGF0IGRvIHRob3NlIG5vdGlmaWVycyBkbz8KClRoYW5r cywKCglJbmdvCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XwprZXhlYyBtYWlsaW5nIGxpc3QKa2V4ZWNAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlz dHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2tleGVjCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752743AbbCWNvA (ORCPT ); Mon, 23 Mar 2015 09:51:00 -0400 Received: from mail-wi0-f173.google.com ([209.85.212.173]:37115 "EHLO mail-wi0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752631AbbCWNuv (ORCPT ); Mon, 23 Mar 2015 09:50:51 -0400 Date: Mon, 23 Mar 2015 14:50:46 +0100 From: Ingo Molnar To: Vivek Goyal Cc: Baoquan He , =?utf-8?B?IkhhdGF5YW1hLCBEYWlzdWtlL+eVkeWxsSDlpKfovJQi?= , ebiederm@xmission.com, masami.hiramatsu.pt@hitachi.com, hidehiro.kawai.ez@hitachi.com, linux-kernel@vger.kernel.org, kexec@lists.infradead.org, akpm@linux-foundation.org, mingo@redhat.com, bp@suse.de Subject: Re: [PATCH v2] kernel/panic/kexec: fix "crash_kexec_post_notifiers" option issue in oops path Message-ID: <20150323135046.GA25012@gmail.com> References: <54F9D645.2050008@jp.fujitsu.com> <20150323034752.GD2068@dhcp-16-105.nay.redhat.com> <20150323071943.GA22765@gmail.com> <20150323133710.GA3172@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20150323133710.GA3172@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Vivek Goyal wrote: > On Mon, Mar 23, 2015 at 08:19:43AM +0100, Ingo Molnar wrote: > > > > * Baoquan He wrote: > > > > > CC more people ... > > > > > > On 03/07/15 at 01:31am, "Hatayama, Daisuke/畑山 大輔" wrote: > > > > The commit f06e5153f4ae2e2f3b0300f0e260e40cb7fefd45 introduced > > > > "crash_kexec_post_notifiers" kernel boot option, which toggles > > > > wheather panic() calls crash_kexec() before panic_notifiers and dump > > > > kmsg or after. > > > > > > > > The problem is that the commit overlooks panic_on_oops kernel boot > > > > option. If it is enabled, crash_kexec() is called directly without > > > > going through panic() in oops path. > > > > > > > > To fix this issue, this patch adds a check to > > > > "crash_kexec_post_notifiers" in the condition of kexec_should_crash(). > > > > > > > > Also, put a comment in kexec_should_crash() to explain not obvious > > > > things on this patch. > > > > > > > > Signed-off-by: HATAYAMA Daisuke > > > > Acked-by: Baoquan He > > > > Tested-by: Hidehiro Kawai > > > > Reviewed-by: Masami Hiramatsu > > > > --- > > > > include/linux/kernel.h | 3 +++ > > > > kernel/kexec.c | 11 +++++++++++ > > > > kernel/panic.c | 2 +- > > > > 3 files changed, 15 insertions(+), 1 deletion(-) > > > > This is hack upon hack, but why was this crap merged in the first > > place? > > > > I see two problems just by cursory review: > > > > 1) > > > > Firstly, the real bug in: > > > > f06e5153f4ae ("kernel/panic.c: add "crash_kexec_post_notifiers" option for kdump after panic_notifers") > > > > Was that crash_kexec() was called unconditionally after notifiers were > > called, which should be fixed via the simple patch below (untested). > > Looks much simpler than your fix. > > > > Hi Ingo, > > Agreed. Your patch looks good. In case you want that simpler fix and need my SOB: Signed-off-by: Ingo Molnar (but I have not tested it.) > > Secondly, and more importantly, the whole premise of commit > > f06e5153f4ae is broken IMHO: > > > > "This can help rare situations where kdump fails because of unstable > > crashed kernel or hardware failure (memory corruption on critical > > data/code)" > > > > wtf? > > > > If the kernel crashed due to a kernel crash, then the kernel booting > > up in whatever hardware state should be able to do a clean bootup. The > > fix for those 'rare situations' should be to fix the real bug (for > > example by making hardware driver init (or deinit) sequences more > > robust), not to paper it over by ordering around crash-time sequences > > ... > > > > If it crashed due to some hardware failure, there's literally an > > infinite amount of failure modes that may or may not be impacted by > > kexec crash-time handling ordering. We don't want to put a zillion > > such flags into the kernel proper just to allow the perturbation of > > the kernel. > > I think one of the motivations behind this patch was call to kmsg_dump(). > Some vendors have been wanting to have the capability to save kernel logs > to some NVRAM before transition to second kernel happens. Their argument > is that kdump does not succeed all the time and if kdump does not succeed > then atleast they have something to work with (kernel logs retrieved > from pstore interface). Doesn't pstore attach itself to printk itself? AFAICS it does: fs/pstore/platform.c: register_console(&pstore_console); so the printk log leading up to and including the crash should be available, regardless of this patch. What am I missing? > Not that I agree fully with this as problem might happen while we > try to run panic_notifiers or kmsg_dump hooks and never transition > into kdump kernel. btw., this is the big problem with 'notifiers' in general: they are opaque with barely any semantics defined, and a source of constant confusion. > And it has been literally years since some developers have been > pushing for allowing to run panic notifiers before crash_kexec(). > Eric Biederman has been pushing back saying it reduces the > reliability of kdump operation so this is not acceptable. So what do those notifiers do? Thanks, Ingo