reboot notifier priority definitions

reboot notifier priority definitions

[Andreas Steinmetz]

As a suggestion to prevent further shutdown/reboot notifier processing problems
here's a (crude) attempt of definitions for include/linux/notifier.h. I do
believe it needs to be heavily reworked by someone with a broad kernel overview.

#define NOTIFY_REBOOT_PHYSICAL    0x00 /* scsi */
#define NOTIFY_REBOOT_LOGICAL     0x10 /* md, lvm */
#define NOTIFY_REBOOT_FS          0x20 /* knfsd */
#define NOTIFY_REBOOT_APPLICATION 0x30 /* tux */

The current situation of not having such definitions already did lead to a
kernel oops (md<->scsi). I do fear that if such definitions are not introduced:

1. code maintainers will strictly stick to priority 0 or will use random
   and not interoperable priorities

2. more and more interference problems will arise due to the above point


Andreas Steinmetz
D.O.M. Datenverarbeitung GmbH

Re: reboot notifier priority definitions

[Neil Brown]

On Tuesday September 11, ast@domdv.de wrote:
> As a suggestion to prevent further shutdown/reboot notifier processing problems
> here's a (crude) attempt of definitions for include/linux/notifier.h. I do
> believe it needs to be heavily reworked by someone with a broad kernel overview.
> 
> #define NOTIFY_REBOOT_PHYSICAL    0x00 /* scsi */
> #define NOTIFY_REBOOT_LOGICAL     0x10 /* md, lvm */
> #define NOTIFY_REBOOT_FS          0x20 /* knfsd */
> #define NOTIFY_REBOOT_APPLICATION 0x30 /* tux */

I think this misses the point of reboot notifiers (as I understand
it).

There are *only* meant for "physical" sorts of things.
The comment in the code says:
 *	Notifier list for kernel code which wants to be called
 *	at shutdown. This is used to stop any idling DMA operations
 *	and the like. 

md, lvm, knfsd and tux have no business registering a reboot notifier.
If they have something to shut down, it should be shut down in a
higher-level way, such as when a process gets a signal. 

Currently md does register a reboot notifier, but this is wrong and
will go away just as soon as it can.  Unfortuantely it cannot yet.
The problem is that if the root filesystem is on an md device, then
the device never received a final close, (bd_op->release) so it can
never make the array as "stable".  That is what the current
reboot notifier does.
I am hoping that when Al Viro's changes to the boot/mount-root sequence
go in, the the root filesystem will really close it's device during
shutdown and the reboot notifier can go away.

Until then, I will change the priority for the md reboot notifier to
be very high, so that it is run first, as you suggest.

I also have this idea that if the raid1d or raid5d gets a signal, it
will switch into a mode where the array is marked stable after each
write, and then marked dirty before each write.  This would have the
desired effect without requiring a final close....

NeilBrown

Re: reboot notifier priority definitions

[Andreas Steinmetz]

> 
> I think this misses the point of reboot notifiers (as I understand
> it).
> 
> There are *only* meant for "physical" sorts of things.
> The comment in the code says:
>  *    Notifier list for kernel code which wants to be called
>  *    at shutdown. This is used to stop any idling DMA operations
>  *    and the like. 
> 
> md, lvm, knfsd and tux have no business registering a reboot notifier.
> If they have something to shut down, it should be shut down in a
> higher-level way, such as when a process gets a signal. 
> 
Even then: My servers do have watchdog cards. Unfortunately without the
priority definitions the watchdog card was shut down prior to the oops. Thus,
due to missing priority, the system did require hitting the reboot button.
So some well defined priorization is still required.


Andreas Steinmetz
D.O.M. Datenverarbeitung GmbH

Re: reboot notifier priority definitions

[Neil Brown]

On Tuesday September 11, ast@domdv.de wrote:
> > 
> > I think this misses the point of reboot notifiers (as I understand
> > it).
> > 
> > There are *only* meant for "physical" sorts of things.
> > The comment in the code says:
> >  *    Notifier list for kernel code which wants to be called
> >  *    at shutdown. This is used to stop any idling DMA operations
> >  *    and the like. 
> > 
> > md, lvm, knfsd and tux have no business registering a reboot notifier.
> > If they have something to shut down, it should be shut down in a
> > higher-level way, such as when a process gets a signal. 
> > 
> Even then: My servers do have watchdog cards. Unfortunately without the
> priority definitions the watchdog card was shut down prior to the oops. Thus,
> due to missing priority, the system did require hitting the reboot button.
> So some well defined priorization is still required.

Fair enough.

Maybe we need one priority for devices,
A lower priority for busses,
An even lower one for watchdogs,

or something like that.
I'll leave to to people who understand hardware configurations and
controllers better than me.

NeilBrown

Re: reboot notifier priority definitions

[Alan Cox]

> Even then: My servers do have watchdog cards. Unfortunately without the
> priority definitions the watchdog card was shut down prior to the oops. Thus,
> due to missing priority, the system did require hitting the reboot button.
> So some well defined priorization is still required.

Its generally a good idea not to use watchdog cards that have shutdown 
sequences - you can easily hang in the BIOS and a reset then may fix the
box.