[PATCH 0/2] add new notifier function

[PATCH 0/2] add new notifier function

[Takenori Nagano]

Hi,

These patches add new notifier function and implement it to panic_notifier_list.
We used the hardcoded notifier chain so far, but it was not flexible. New
notifier is very flexible, because user can change a list of order by debugfs.

Please review, and give some comments.

Thanks,

Example)

# cd /sys/kernel/debug/
# ls
kprobes  pktcdvd
# insmod ipmi_msghandler.ko
# ls
kprobes  panic_notifier_list  pktcdvd
# cd panic_notifier_list/
# ls
ipmi_msghandler
# insmod ipmi_watchdog.ko
# ls
ipmi_msghandler  ipmi_wdog
# cat ipmi_msghandler/priority
200
# cat ipmi_wdog/priority
150
#
Kernel panic - not syncing: panic
ipmi_msghandler : notifier calls panic_event().
ipmi_watchdog : notifier calls wdog_panic_handler().

.....(reboot)

# cat ipmi_msghandler/priority
200
# cat ipmi_wdog/priority
150
# echo 300 > ipmi_wdog/priority
#
Kernel panic - not syncing: panic
ipmi_watchdog : notifier calls wdog_panic_handler().
ipmi_msghandler : notifier calls panic_event().

-- 
Takenori Nagano <t-nagano@ah.jp.nec.com>



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 0/2] add new notifier function

[Takenori Nagano]

Hi,

These patches add new notifier function and implement it to panic_notifier_list.
We used the hardcoded notifier chain so far, but it was not flexible. New
notifier is very flexible, because user can change a list of order by debugfs.

Please review, and give some comments.

Thanks,

Example)

# cd /sys/kernel/debug/
# ls
kprobes  pktcdvd
# insmod ipmi_msghandler.ko
# ls
kprobes  panic_notifier_list  pktcdvd
# cd panic_notifier_list/
# ls
ipmi_msghandler
# insmod ipmi_watchdog.ko
# ls
ipmi_msghandler  ipmi_wdog
# cat ipmi_msghandler/priority
200
# cat ipmi_wdog/priority
150
#
Kernel panic - not syncing: panic
ipmi_msghandler : notifier calls panic_event().
ipmi_watchdog : notifier calls wdog_panic_handler().

.....(reboot)

# cat ipmi_msghandler/priority
200
# cat ipmi_wdog/priority
150
# echo 300 > ipmi_wdog/priority
#
Kernel panic - not syncing: panic
ipmi_watchdog : notifier calls wdog_panic_handler().
ipmi_msghandler : notifier calls panic_event().

-- 
Takenori Nagano <t-nagano@ah.jp.nec.com>

Re: [PATCH 0/2] add new notifier function

[Vivek Goyal]

On Thu, Oct 04, 2007 at 08:38:05PM +0900, Takenori Nagano wrote:
> Hi,
> 
> These patches add new notifier function and implement it to panic_notifier_list.
> We used the hardcoded notifier chain so far, but it was not flexible. New
> notifier is very flexible, because user can change a list of order by debugfs.
> 

Hi Takenori,

There were some more discussions regarding configurable notifier list.
Following is the link. Please go through it.

http://marc.info/?l=linux-kernel&m=118968996202991&w=2

Not everybody is too happy about it. Personally I am not against it. My take
is that after panic() there is no gurantee that all the registered notifer
will be executed. Just that kernel will try its best. If a notifier handler
is written badly, kernel can't do much about it. It is left more on to
administrator what he considers most important and give priority accordingly.

So if kdump is of utmost priority, then administrator should give highest
priority to kdump. 

Having said that, what are the RAS tools which require this infrastructure.
Currently only kdb seems to be the only candidate which needs to run in
the crashing kernel. Rest of the actions can be performed in second kernel.
If that is the case, then probably it is better that kdb puts a break point
on panic(), as suggested by Eric, and rest of the post panic actions are
executed in second kernel.

Executing rest of the actions have got both pros and cons. Executing rest
of the notifications in second kernel makes things more reliable. At the same
time it makes things little complex as one needs to pass all the configuration
information required to second kernel, secondly all the notification handlers
need to be ready to run in two contexts. These handlers will run in the 
context of first kernel if kdump is not configured, otherwise these will need
to run in second kernel.

In summary, right now co-existence of kdb with kdump seems to be your pain
point. I would prefer that kdb just puts a break point on panic() and we move
on. If there are more candidates down the line and these can't be easily
executed in second kernel then we can re-visit this notification list
mechanism.
   
 
> Please review, and give some comments.
> 
> Thanks,
> 
> Example)
> 
> # cd /sys/kernel/debug/
> # ls
> kprobes  pktcdvd
> # insmod ipmi_msghandler.ko
> # ls
> kprobes  panic_notifier_list  pktcdvd
> # cd panic_notifier_list/
> # ls
> ipmi_msghandler
> # insmod ipmi_watchdog.ko
> # ls
> ipmi_msghandler  ipmi_wdog
> # cat ipmi_msghandler/priority
> 200
> # cat ipmi_wdog/priority
> 150
> #
> Kernel panic - not syncing: panic
> ipmi_msghandler : notifier calls panic_event().
> ipmi_watchdog : notifier calls wdog_panic_handler().
> 
> .....(reboot)
> 

We also need to implement a file which can give a consolidated view. All
the registered members and their priority.

Thanks
Vivek

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] add new notifier function

[Vivek Goyal]

On Thu, Oct 04, 2007 at 08:38:05PM +0900, Takenori Nagano wrote:
> Hi,
> 
> These patches add new notifier function and implement it to panic_notifier_list.
> We used the hardcoded notifier chain so far, but it was not flexible. New
> notifier is very flexible, because user can change a list of order by debugfs.
> 

Hi Takenori,

There were some more discussions regarding configurable notifier list.
Following is the link. Please go through it.

http://marc.info/?l=linux-kernel&m=118968996202991&w=2

Not everybody is too happy about it. Personally I am not against it. My take
is that after panic() there is no gurantee that all the registered notifer
will be executed. Just that kernel will try its best. If a notifier handler
is written badly, kernel can't do much about it. It is left more on to
administrator what he considers most important and give priority accordingly.

So if kdump is of utmost priority, then administrator should give highest
priority to kdump. 

Having said that, what are the RAS tools which require this infrastructure.
Currently only kdb seems to be the only candidate which needs to run in
the crashing kernel. Rest of the actions can be performed in second kernel.
If that is the case, then probably it is better that kdb puts a break point
on panic(), as suggested by Eric, and rest of the post panic actions are
executed in second kernel.

Executing rest of the actions have got both pros and cons. Executing rest
of the notifications in second kernel makes things more reliable. At the same
time it makes things little complex as one needs to pass all the configuration
information required to second kernel, secondly all the notification handlers
need to be ready to run in two contexts. These handlers will run in the 
context of first kernel if kdump is not configured, otherwise these will need
to run in second kernel.

In summary, right now co-existence of kdb with kdump seems to be your pain
point. I would prefer that kdb just puts a break point on panic() and we move
on. If there are more candidates down the line and these can't be easily
executed in second kernel then we can re-visit this notification list
mechanism.
   
 
> Please review, and give some comments.
> 
> Thanks,
> 
> Example)
> 
> # cd /sys/kernel/debug/
> # ls
> kprobes  pktcdvd
> # insmod ipmi_msghandler.ko
> # ls
> kprobes  panic_notifier_list  pktcdvd
> # cd panic_notifier_list/
> # ls
> ipmi_msghandler
> # insmod ipmi_watchdog.ko
> # ls
> ipmi_msghandler  ipmi_wdog
> # cat ipmi_msghandler/priority
> 200
> # cat ipmi_wdog/priority
> 150
> #
> Kernel panic - not syncing: panic
> ipmi_msghandler : notifier calls panic_event().
> ipmi_watchdog : notifier calls wdog_panic_handler().
> 
> .....(reboot)
> 

We also need to implement a file which can give a consolidated view. All
the registered members and their priority.

Thanks
Vivek

Re: [PATCH 0/2] add new notifier function

[Takenori Nagano]

Vivek Goyal wrote:
> On Thu, Oct 04, 2007 at 08:38:05PM +0900, Takenori Nagano wrote:
>
> In summary, right now co-existence of kdb with kdump seems to be your pain
> point. I would prefer that kdb just puts a break point on panic() and we move
> on. If there are more candidates down the line and these can't be easily
> executed in second kernel then we can re-visit this notification list
> mechanism.

Hi Vivek,

Thank you for your comment. :-)

I don't mind kdb and kdump problem now. Because my patches are not merged into
mainline kernel yet. If they are merged, I think how we can resolve about RAS
tools problem.

>> # ls
>> ipmi_msghandler  ipmi_wdog
>> # cat ipmi_msghandler/priority
>> 200
>> # cat ipmi_wdog/priority
>> 150
>> #
>> Kernel panic - not syncing: panic
>> ipmi_msghandler : notifier calls panic_event().
>> ipmi_watchdog : notifier calls wdog_panic_handler().
>>
>> .....(reboot)
>>
> 
> We also need to implement a file which can give a consolidated view. All
> the registered members and their priority.

I tried to implement it, but its impact is large. And we can get all priority
values using "ls" and "cat */priority". I'll implement it if user strongly
expects it.

ex)
# cd panic_notifier_list
# ls
ipmi_msghandler  ipmi_wdog
# cat */priority
200
150
#

Thanks,

Takenori Nagano <t-nagano@ah.jp.nec.com>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] add new notifier function

[Takenori Nagano]

Vivek Goyal wrote:
> On Thu, Oct 04, 2007 at 08:38:05PM +0900, Takenori Nagano wrote:
>
> In summary, right now co-existence of kdb with kdump seems to be your pain
> point. I would prefer that kdb just puts a break point on panic() and we move
> on. If there are more candidates down the line and these can't be easily
> executed in second kernel then we can re-visit this notification list
> mechanism.

Hi Vivek,

Thank you for your comment. :-)

I don't mind kdb and kdump problem now. Because my patches are not merged into
mainline kernel yet. If they are merged, I think how we can resolve about RAS
tools problem.

>> # ls
>> ipmi_msghandler  ipmi_wdog
>> # cat ipmi_msghandler/priority
>> 200
>> # cat ipmi_wdog/priority
>> 150
>> #
>> Kernel panic - not syncing: panic
>> ipmi_msghandler : notifier calls panic_event().
>> ipmi_watchdog : notifier calls wdog_panic_handler().
>>
>> .....(reboot)
>>
> 
> We also need to implement a file which can give a consolidated view. All
> the registered members and their priority.

I tried to implement it, but its impact is large. And we can get all priority
values using "ls" and "cat */priority". I'll implement it if user strongly
expects it.

ex)
# cd panic_notifier_list
# ls
ipmi_msghandler  ipmi_wdog
# cat */priority
200
150
#

Thanks,

Takenori Nagano <t-nagano@ah.jp.nec.com>

Re: [PATCH 0/2] add new notifier function

[Eric W. Biederman]

Takenori Nagano <t-nagano@ah.jp.nec.com> writes:

> Hi,
>
> These patches add new notifier function and implement it to panic_notifier_list.
> We used the hardcoded notifier chain so far, but it was not flexible. New
> notifier is very flexible, because user can change a list of order by debugfs.

How is the lack of flexibility a problem?
Specifics please.

My impression is that the purpose of this patchset is to build
infrastructure to sort out a conflict between kdb and the kexec code,
which it does not do, and it can not do if it does not own up to
it's real purpose.

If I am correct in understanding the purpose of this patchset it does
not even address the problem it is aimed at solving.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] add new notifier function

[Eric W. Biederman]

Takenori Nagano <t-nagano@ah.jp.nec.com> writes:

> Hi,
>
> These patches add new notifier function and implement it to panic_notifier_list.
> We used the hardcoded notifier chain so far, but it was not flexible. New
> notifier is very flexible, because user can change a list of order by debugfs.

How is the lack of flexibility a problem?
Specifics please.

My impression is that the purpose of this patchset is to build
infrastructure to sort out a conflict between kdb and the kexec code,
which it does not do, and it can not do if it does not own up to
it's real purpose.

If I am correct in understanding the purpose of this patchset it does
not even address the problem it is aimed at solving.

Eric

Re: [PATCH 0/2] add new notifier function

[Takenori Nagano]

Eric W. Biederman wrote:
> Takenori Nagano <t-nagano@ah.jp.nec.com> writes:
> 
>> Hi,
>>
>> These patches add new notifier function and implement it to panic_notifier_list.
>> We used the hardcoded notifier chain so far, but it was not flexible. New
>> notifier is very flexible, because user can change a list of order by debugfs.
> 
> How is the lack of flexibility a problem?
> Specifics please.

Please read this again.
http://www.gossamer-threads.com/lists/linux/kernel/797220?do=post_view_threaded#797220

Keith Owen said,

> My stance is that _all_ the RAS tools (kdb, kgdb, nlkd, netdump, lkcd,
> crash, kdump etc.) should be using a common interface that safely puts
> the entire system in a stopped state and saves the state of each cpu.
> Then each tool can do what it likes, instead of every RAS tool doing
> its own thing and they all conflict with each other, which is why this
> thread started.
> 
> It is not the kernel's job to decide which RAS tool runs first, second
> etc., it is the user's decision to set that policy. Different sites
> will want different orders, some will say "go straight to kdump", other
> sites will want to invoke a debugger first. Sites must be able to
> define that policy, but we hard code the policy into the kernel. 

I agreed with him and I made new notifier function.

> 
> My impression is that the purpose of this patchset is to build
> infrastructure to sort out a conflict between kdb and the kexec code,
> which it does not do, and it can not do if it does not own up to
> it's real purpose.

My motivation does not change. But I don't think kdump have to use notifer.
I want to resolve this adopting the way which satisfy all users.

Thanks,

Takenori Nagano <t-nagano@ah.jp.nec.com>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/2] add new notifier function

[Takenori Nagano]

Eric W. Biederman wrote:
> Takenori Nagano <t-nagano@ah.jp.nec.com> writes:
> 
>> Hi,
>>
>> These patches add new notifier function and implement it to panic_notifier_list.
>> We used the hardcoded notifier chain so far, but it was not flexible. New
>> notifier is very flexible, because user can change a list of order by debugfs.
> 
> How is the lack of flexibility a problem?
> Specifics please.

Please read this again.
http://www.gossamer-threads.com/lists/linux/kernel/797220?do=post_view_threaded#797220

Keith Owen said,

> My stance is that _all_ the RAS tools (kdb, kgdb, nlkd, netdump, lkcd,
> crash, kdump etc.) should be using a common interface that safely puts
> the entire system in a stopped state and saves the state of each cpu.
> Then each tool can do what it likes, instead of every RAS tool doing
> its own thing and they all conflict with each other, which is why this
> thread started.
> 
> It is not the kernel's job to decide which RAS tool runs first, second
> etc., it is the user's decision to set that policy. Different sites
> will want different orders, some will say "go straight to kdump", other
> sites will want to invoke a debugger first. Sites must be able to
> define that policy, but we hard code the policy into the kernel. 

I agreed with him and I made new notifier function.

> 
> My impression is that the purpose of this patchset is to build
> infrastructure to sort out a conflict between kdb and the kexec code,
> which it does not do, and it can not do if it does not own up to
> it's real purpose.

My motivation does not change. But I don't think kdump have to use notifer.
I want to resolve this adopting the way which satisfy all users.

Thanks,

Takenori Nagano <t-nagano@ah.jp.nec.com>