All of lore.kernel.org
 help / color / mirror / Atom feed
From: Denys Fedoryshchenko <nuclearcat@nuclearcat.com>
To: Jon Masters <jcm@jonmasters.org>
Cc: kexec@lists.infradead.org, linux-kernel@vger.kernel.org
Subject: Re: kexec on panic
Date: Sat, 18 Feb 2017 10:09:39 +0200	[thread overview]
Message-ID: <8dee042080bf3a00d376afc271b050eb@nuclearcat.com> (raw)
In-Reply-To: <23681d9b-fec9-8ec7-3772-db5b36da9725@jonmasters.org>

On 2017-02-18 09:42, Jon Masters wrote:
> Hi Denys,
> 
> On 02/10/2017 03:14 AM, Denys Fedoryshchenko wrote:
> 
>> After years of using kexec and recent unpleasant experience with 
>> modern (supposed to be blazing fast to boot) hardware that need 5-10 
>> minutes just to pass POST tests,
>> one question came up to me:
>> Is it possible anyhow to execute regular (not special "panic" one to 
>> capture crash data) kexec on panic to reduce reboot time?
> 
> Generally, you don't want to do this, because various platform hardware
> might be in non-quiescent states (still doing DMA to random memory, 
> etc.)
> and other nastiness that means you don't want to do more than the 
> minimal
> amount in a kexec on panic (crash). We've seen no end of fun and games
> even with just regular crash dumps while hardware is busily writing to
> memory that it shouldn't be. An IOMMU helps, but isn't a cure-all.
> 
> Jon.
Well, i have to try, even sometimes i am facing issues with non-booting 
hardware even on regular kexec, but having at small customer HP server 
that need almost 6 minutes to boot,
no hot-spare(and hard to do by many reasons, no spare 10G ports, cost of 
hardware and etc) and some nasty bugs that is not resolved yet - forcing 
me to search way to reduce reboot time.
If i will find way to save backtrace and reboot fast, it will help a lot 
to debug kernels with minimal downtime, if bug is reproducible only on 
live system.

What i did now, might be insanely wrong, but:
diff -Naur linux-4.9.9-vanilla/kernel/kexec_core.c 
linux-4.9.9/kernel/kexec_core.c
--- linux-4.9.9-vanilla/kernel/kexec_core.c	2017-02-09 
07:08:40.000000000 +0000
+++ linux-4.9.9/kernel/kexec_core.c	2017-02-17 12:54:49.000000000 +0000
@@ -897,6 +897,10 @@
  			machine_crash_shutdown(&fixed_regs);
  			machine_kexec(kexec_crash_image);
  		}
+		if (kexec_image) {
+		    machine_shutdown();
+		    machine_kexec(kexec_image);
+		}
  		mutex_unlock(&kexec_mutex);
  	}
  }

Then

kexec -l /mnt/flash/kernel --append="intel_idle.max_cstate=0 
processor.max_cstate=1"

and
echo c >/proc/sysrq-trigger
worked even on busy network router, but i'm not sure it will be same on 
real networking stack crash.


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

WARNING: multiple messages have this Message-ID (diff)
From: Denys Fedoryshchenko <nuclearcat@nuclearcat.com>
To: Jon Masters <jcm@jonmasters.org>
Cc: linux-kernel@vger.kernel.org, kexec@lists.infradead.org
Subject: Re: kexec on panic
Date: Sat, 18 Feb 2017 10:09:39 +0200	[thread overview]
Message-ID: <8dee042080bf3a00d376afc271b050eb@nuclearcat.com> (raw)
In-Reply-To: <23681d9b-fec9-8ec7-3772-db5b36da9725@jonmasters.org>

On 2017-02-18 09:42, Jon Masters wrote:
> Hi Denys,
> 
> On 02/10/2017 03:14 AM, Denys Fedoryshchenko wrote:
> 
>> After years of using kexec and recent unpleasant experience with 
>> modern (supposed to be blazing fast to boot) hardware that need 5-10 
>> minutes just to pass POST tests,
>> one question came up to me:
>> Is it possible anyhow to execute regular (not special "panic" one to 
>> capture crash data) kexec on panic to reduce reboot time?
> 
> Generally, you don't want to do this, because various platform hardware
> might be in non-quiescent states (still doing DMA to random memory, 
> etc.)
> and other nastiness that means you don't want to do more than the 
> minimal
> amount in a kexec on panic (crash). We've seen no end of fun and games
> even with just regular crash dumps while hardware is busily writing to
> memory that it shouldn't be. An IOMMU helps, but isn't a cure-all.
> 
> Jon.
Well, i have to try, even sometimes i am facing issues with non-booting 
hardware even on regular kexec, but having at small customer HP server 
that need almost 6 minutes to boot,
no hot-spare(and hard to do by many reasons, no spare 10G ports, cost of 
hardware and etc) and some nasty bugs that is not resolved yet - forcing 
me to search way to reduce reboot time.
If i will find way to save backtrace and reboot fast, it will help a lot 
to debug kernels with minimal downtime, if bug is reproducible only on 
live system.

What i did now, might be insanely wrong, but:
diff -Naur linux-4.9.9-vanilla/kernel/kexec_core.c 
linux-4.9.9/kernel/kexec_core.c
--- linux-4.9.9-vanilla/kernel/kexec_core.c	2017-02-09 
07:08:40.000000000 +0000
+++ linux-4.9.9/kernel/kexec_core.c	2017-02-17 12:54:49.000000000 +0000
@@ -897,6 +897,10 @@
  			machine_crash_shutdown(&fixed_regs);
  			machine_kexec(kexec_crash_image);
  		}
+		if (kexec_image) {
+		    machine_shutdown();
+		    machine_kexec(kexec_image);
+		}
  		mutex_unlock(&kexec_mutex);
  	}
  }

Then

kexec -l /mnt/flash/kernel --append="intel_idle.max_cstate=0 
processor.max_cstate=1"

and
echo c >/proc/sysrq-trigger
worked even on busy network router, but i'm not sure it will be same on 
real networking stack crash.

  reply	other threads:[~2017-02-18  8:10 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-02-10  8:14 kexec on panic Denys Fedoryshchenko
2017-02-10  8:14 ` Denys Fedoryshchenko
2017-02-10 15:43 ` Petr Tesarik
2017-02-15 17:29   ` Clif Houck
2017-02-15 21:45     ` Marc Milgram
2017-02-18  7:42 ` Jon Masters
2017-02-18  7:42   ` Jon Masters
2017-02-18  8:09   ` Denys Fedoryshchenko [this message]
2017-02-18  8:09     ` Denys Fedoryshchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8dee042080bf3a00d376afc271b050eb@nuclearcat.com \
    --to=nuclearcat@nuclearcat.com \
    --cc=jcm@jonmasters.org \
    --cc=kexec@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.