watchdogs and kdump

watchdogs and kdump

[Don Zickus]

Hi,

I was assisting a customer the other day debugging a kdump[1] problem, when we
noticed the real problem was the hardware watchdog was firing and
rebooting the box.

Of course, this can be inconvienant if the panic happens right before the
watchdog is supposed to be kicked, leading to a spontaneous reboot before
the second kernel finishes booting and loading the watchdog module.

I was trying to think of a way to solve this and thought, one way to
minimize the problem is to kick the watchdog before we jump into the kdump
kernel.  Another way is to disable the watchdog entirely, but that doesn't
work on all hardware I believe.

Anyway, I was posting on the watchdog mailing list to see if anyone had any
ideas that might help.  And if my above idea to kick the watchdog before
jumping into the kdump kernel seems ok, then an api would need to be
developed.

I am willing to do any coding and testing necessary, but before I did, I
wanted help to get a direction to go in first.

Thoughts?

Cheers,
Don

[1] - I am ignorantly assuming everyone knows what kdump is.  Kdumping is
the ability to jump into a previously loaded kernel in the case of a
panic.  This kdump (second) kernel would run in reserved memory, copy the
first kernel's memory to a file and save it to a pre-determined location.
There is no system reboot in between the first and second kernel, so no
chance for the watchdog to disarm itself.

Re: watchdogs and kdump

[Pádraig Brady]

On 10/27/2011 09:30 PM, Don Zickus wrote:
> Hi,
> 
> I was assisting a customer the other day debugging a kdump[1] problem, when we
> noticed the real problem was the hardware watchdog was firing and
> rebooting the box.
> 
> Of course, this can be inconvienant if the panic happens right before the
> watchdog is supposed to be kicked, leading to a spontaneous reboot before
> the second kernel finishes booting and loading the watchdog module.
> 
> I was trying to think of a way to solve this and thought, one way to
> minimize the problem is to kick the watchdog before we jump into the kdump
> kernel.  Another way is to disable the watchdog entirely, but that doesn't
> work on all hardware I believe.
> 
> Anyway, I was posting on the watchdog mailing list to see if anyone had any
> ideas that might help.  And if my above idea to kick the watchdog before
> jumping into the kdump kernel seems ok, then an api would need to be
> developed.
> 
> I am willing to do any coding and testing necessary, but before I did, I
> wanted help to get a direction to go in first.
> 
> Thoughts?

Seems like the appropriate thing to do is to call all the
reboot notifiers that each watchdog registers.
Since one is not doingn a full SYS_RESTART (SYS_DOWN) though,
i.e. not running through the BIOS code again,
it might be worth having a different SYS_JUMP code in notifier.h
that would allow you to kick rather than stop the watchdogs
as the reboot notifiers generally do at the moment.
I think it would be important not to stop the watchdog if possible,
given the large amount of logic that's going to be executed
after the jump.

cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-watchdog" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: watchdogs and kdump

[Don Zickus]

On Thu, Oct 27, 2011 at 10:43:58PM +0100, Pádraig Brady wrote:
> On 10/27/2011 09:30 PM, Don Zickus wrote:
> > Hi,
> > 
> > I was assisting a customer the other day debugging a kdump[1] problem, when we
> > noticed the real problem was the hardware watchdog was firing and
> > rebooting the box.
> > 
> > Of course, this can be inconvienant if the panic happens right before the
> > watchdog is supposed to be kicked, leading to a spontaneous reboot before
> > the second kernel finishes booting and loading the watchdog module.
> > 
> > I was trying to think of a way to solve this and thought, one way to
> > minimize the problem is to kick the watchdog before we jump into the kdump
> > kernel.  Another way is to disable the watchdog entirely, but that doesn't
> > work on all hardware I believe.
> > 
> > Anyway, I was posting on the watchdog mailing list to see if anyone had any
> > ideas that might help.  And if my above idea to kick the watchdog before
> > jumping into the kdump kernel seems ok, then an api would need to be
> > developed.
> > 
> > I am willing to do any coding and testing necessary, but before I did, I
> > wanted help to get a direction to go in first.
> > 
> > Thoughts?
> 
> Seems like the appropriate thing to do is to call all the
> reboot notifiers that each watchdog registers.
> Since one is not doingn a full SYS_RESTART (SYS_DOWN) though,
> i.e. not running through the BIOS code again,
> it might be worth having a different SYS_JUMP code in notifier.h
> that would allow you to kick rather than stop the watchdogs
> as the reboot notifiers generally do at the moment.

That is an interesting idea.  Not sure if calling a blocking notifier in
the kdump path would be acceptable to the kexec folks.  Then again using
the reboot notifier in the panic path may not be a good idea either, it
might lead to false expectations. :-/

> I think it would be important not to stop the watchdog if possible,
> given the large amount of logic that's going to be executed
> after the jump.

I agree.  Especially since kdump is still not 100% reliable.

Thanks for the feedback!

Cheers,
Don
--
To unsubscribe from this list: send the line "unsubscribe linux-watchdog" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: watchdogs and kdump

[Alejandro Cabrera]

Hi

I dont know kjump :), but seeing it's description I think that you could 
use a temporal thread executed in the context of kdump that ping the 
watchdog at certain intervals like watchdogd does at user-space.

Regards
Alejandro


On 10/27/2011 4:30 PM, Don Zickus wrote:
> Hi,
>
> I was assisting a customer the other day debugging a kdump[1] problem, when we
> noticed the real problem was the hardware watchdog was firing and
> rebooting the box.
>
> Of course, this can be inconvienant if the panic happens right before the
> watchdog is supposed to be kicked, leading to a spontaneous reboot before
> the second kernel finishes booting and loading the watchdog module.
>
> I was trying to think of a way to solve this and thought, one way to
> minimize the problem is to kick the watchdog before we jump into the kdump
> kernel.  Another way is to disable the watchdog entirely, but that doesn't
> work on all hardware I believe.
>
> Anyway, I was posting on the watchdog mailing list to see if anyone had any
> ideas that might help.  And if my above idea to kick the watchdog before
> jumping into the kdump kernel seems ok, then an api would need to be
> developed.
>
> I am willing to do any coding and testing necessary, but before I did, I
> wanted help to get a direction to go in first.
>
> Thoughts?
>
> Cheers,
> Don
>
> [1] - I am ignorantly assuming everyone knows what kdump is.  Kdumping is
> the ability to jump into a previously loaded kernel in the case of a
> panic.  This kdump (second) kernel would run in reserved memory, copy the
> first kernel's memory to a file and save it to a pre-determined location.
> There is no system reboot in between the first and second kernel, so no
> chance for the watchdog to disarm itself.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-watchdog" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Participe en Universidad 2012, del 13 al 17 de febrero de 2012.
Habana, Cuba: http://www.congresouniversidad.cu
Consulte la enciclopedia colaborativa cubana. http://www.ecured.cu

Participe en el Segundo Congreso Medio Ambiente Construido y 
Desarrollo Sustentable (MACDES 2011) del 6 al 9 de diciembre de 2011, 
Hotel Nacional, Habana, Cuba: http://macdes.cujae.edu.cu

Re: watchdogs and kdump

[Don Zickus]

On Fri, Oct 28, 2011 at 11:46:30AM -0400, Alejandro Cabrera wrote:
> Hi
> 
> I dont know kjump :), but seeing it's description I think that you
> could use a temporal thread executed in the context of kdump that
> ping the watchdog at certain intervals like watchdogd does at
> user-space.

Sure.  Add something like watchdogd isn't difficult.  My problem is
getting enough time to boot the second kernel to run that daemon.
Depending on when the watchdog was last kicked, the machine may reboot
while trying to initialize the cpu in the second kernel. :-(

Cheers,
Don

Re: watchdogs and kdump

[Alejandro Cabrera]

On 10/28/2011 11:48 AM, Don Zickus wrote:
> On Fri, Oct 28, 2011 at 11:46:30AM -0400, Alejandro Cabrera wrote:
>> Hi
>>
>> I dont know kjump :), but seeing it's description I think that you
>> could use a temporal thread executed in the context of kdump that
>> ping the watchdog at certain intervals like watchdogd does at
>> user-space.
> Sure.  Add something like watchdogd isn't difficult.  My problem is
> getting enough time to boot the second kernel to run that daemon.
> Depending on when the watchdog was last kicked, the machine may reboot
> while trying to initialize the cpu in the second kernel. :-(

You can create the daemon in kdump before the second kernel boots and 
manage that daemon as a kdump thread.

While the second kernel is booting kdump will act as the owner of the 
watchdog device and when the second kernel is stable, kdump passes the 
control  of it to the watchdog kernel device driver.
If is not problem, that two threads (kdump daemon and second kernel 
watchdog driver) access to the device simultaneously and without sync (I 
think IMMO that for pinging the watchdog it is not a issue) you can 
bypass the part that second kernel wdt drivers waits a notification from 
kdump to start working with the watchdog device.

Regards
Alejandro


Participe en Universidad 2012, del 13 al 17 de febrero de 2012.
Habana, Cuba: http://www.congresouniversidad.cu
Consulte la enciclopedia colaborativa cubana. http://www.ecured.cu

Participe en el Segundo Congreso Medio Ambiente Construido y 
Desarrollo Sustentable (MACDES 2011) del 6 al 9 de diciembre de 2011, 
Hotel Nacional, Habana, Cuba: http://macdes.cujae.edu.cu

Re: watchdogs and kdump

[Don Zickus]

On Fri, Oct 28, 2011 at 12:13:06PM -0400, Alejandro Cabrera wrote:
> On 10/28/2011 11:48 AM, Don Zickus wrote:
> >On Fri, Oct 28, 2011 at 11:46:30AM -0400, Alejandro Cabrera wrote:
> >>Hi
> >>
> >>I dont know kjump :), but seeing it's description I think that you
> >>could use a temporal thread executed in the context of kdump that
> >>ping the watchdog at certain intervals like watchdogd does at
> >>user-space.
> >Sure.  Add something like watchdogd isn't difficult.  My problem is
> >getting enough time to boot the second kernel to run that daemon.
> >Depending on when the watchdog was last kicked, the machine may reboot
> >while trying to initialize the cpu in the second kernel. :-(
> 
> You can create the daemon in kdump before the second kernel boots
> and manage that daemon as a kdump thread.

Unfortunately, you can't do that as only one kernel runs at any given
time.  When the first kernel panics, it does some quick and dirty
shutdowns of platform specific chips, forces all the cpus into a halt
state and then longjmps into the second kernel.

Cheers,
Don

> 
> While the second kernel is booting kdump will act as the owner of
> the watchdog device and when the second kernel is stable, kdump
> passes the control  of it to the watchdog kernel device driver.
> If is not problem, that two threads (kdump daemon and second kernel
> watchdog driver) access to the device simultaneously and without
> sync (I think IMMO that for pinging the watchdog it is not a issue)
> you can bypass the part that second kernel wdt drivers waits a
> notification from kdump to start working with the watchdog device.
> 
> Regards
> Alejandro
> 
> 
> Participe en Universidad 2012, del 13 al 17 de febrero de 2012.
> Habana, Cuba: http://www.congresouniversidad.cu
> Consulte la enciclopedia colaborativa cubana. http://www.ecured.cu
> 
> Participe en el Segundo Congreso Medio Ambiente Construido y
> Desarrollo Sustentable (MACDES 2011) del 6 al 9 de diciembre de
> 2011, Hotel Nacional, Habana, Cuba: http://macdes.cujae.edu.cu