/var/log/messages doesn't have crash info when kernel gets panic/oops/crash

/var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

Hi,

I have encounter a problem called /var/log/messages does not contain any information 
on kernel crash dump.

Prior to enabling KEXEC we saw stack traces and bug info etc in /var/log/messages.
After enabling KEXEC feature those info was missed.

How can we get those missed info in /var/log/messages even when KEXEC feature is enabled.

Any help will appreciate.


Thanks and Regards
Seetharam

____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Neil Horman]

On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
> Hi,
> 
> I have encounter a problem called /var/log/messages does not contain any information 
> on kernel crash dump.
> 
> Prior to enabling KEXEC we saw stack traces and bug info etc in /var/log/messages.
> After enabling KEXEC feature those info was missed.
> 
> How can we get those missed info in /var/log/messages even when KEXEC feature is enabled.
> 
> Any help will appreciate.
> 
run crash on the resultant vmcore file that kexec can produce/save and get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log

Neil

> 
> Thanks and Regards
> Seetharam
> 
> ____________________________________________________________
> Receive Notifications of Incoming Messages
> Easily monitor multiple email accounts & access them with a click.
> Visit http://www.inbox.com/notifier and check it out!
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

Hi Neil,

Thanks for your quick reply.

What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted etc. then we dont get any information
regarding the panic/crash/oops at anywhere?

 


Thanks and Regards

 Seetharam


> -----Original Message-----
> From: nhorman@redhat.com
> Sent: Fri, 12 Jun 2009 16:05:13 -0400
> To: dseetharam@inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
> 
> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
>> Hi,
>> 
>> I have encounter a problem called /var/log/messages does not contain any
>> information
>> on kernel crash dump.
>> 
>> Prior to enabling KEXEC we saw stack traces and bug info etc in
>> /var/log/messages.
>> After enabling KEXEC feature those info was missed.
>> 
>> How can we get those missed info in /var/log/messages even when KEXEC
>> feature is enabled.
>> 
>> Any help will appreciate.
>> 
> run crash on the resultant vmcore file that kexec can produce/save and
> get the
> dmesg log out of it.
> alternatively you can use makedumpfile with the --dump-dmesg log
> 
> Neil
> 
>> 
>> Thanks and Regards
>> Seetharam
>> 
>> ____________________________________________________________
>> Receive Notifications of Incoming Messages
>> Easily monitor multiple email accounts & access them with a click.
>> Visit http://www.inbox.com/notifier and check it out!
>> 
>> _______________________________________________
>> kexec mailing list
>> kexec@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/kexec

____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at http://www.inbox.com/smileys
Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™ and most webmails

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Eric W. Biederman]

Dharmosoth Seetharam <dseetharam@inbox.com> writes:

> Hi Neil,
>
> Thanks for your quick reply.
>
> What you have suggested is nice when we got/save crash dump (i.e vmcore)
> What if we failed to save/capture vmcore info or vmcore is corrupted etc. then we dont get any information
> regarding the panic/crash/oops at anywhere?

Basic questions.
1) Do you have panic on oops set?
   I think that setting more than anything else will be the
   difference in what shows up in /var/log/messages.
2) Have you tried a dry run and confirmed you can get a crash dump?
3) Are you using kvm


Eric


> Thanks and Regards
>
>  Seetharam
>
>
>> -----Original Message-----
>> From: nhorman@redhat.com
>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
>> To: dseetharam@inbox.com
>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>> panic/oops/crash
>> 
>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
>>> Hi,
>>> 
>>> I have encounter a problem called /var/log/messages does not contain any
>>> information
>>> on kernel crash dump.
>>> 
>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
>>> /var/log/messages.
>>> After enabling KEXEC feature those info was missed.
>>> 
>>> How can we get those missed info in /var/log/messages even when KEXEC
>>> feature is enabled.
>>> 
>>> Any help will appreciate.
>>> 
>> run crash on the resultant vmcore file that kexec can produce/save and
>> get the
>> dmesg log out of it.
>> alternatively you can use makedumpfile with the --dump-dmesg log
>> 
>> Neil
>> 
>>> 
>>> Thanks and Regards
>>> Seetharam
>>> 
>>> ____________________________________________________________
>>> Receive Notifications of Incoming Messages
>>> Easily monitor multiple email accounts & access them with a click.
>>> Visit http://www.inbox.com/notifier and check it out!
>>> 
>>> _______________________________________________
>>> kexec mailing list
>>> kexec@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/kexec
>
> ____________________________________________________________
> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at http://www.inbox.com/smileys
> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™ and most webmails

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

> -----Original Message-----
> From: ebiederm@xmission.com
> Sent: Sat, 13 Jun 2009 00:11:25 -0700
> To: dseetharam@inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
> 
> Dharmosoth Seetharam <dseetharam@inbox.com> writes:
> 
>> Hi Neil,
>> 
>> Thanks for your quick reply.
>> 
>> What you have suggested is nice when we got/save crash dump (i.e vmcore)
>> What if we failed to save/capture vmcore info or vmcore is corrupted
>> etc. then we dont get any information
>> regarding the panic/crash/oops at anywhere?
> 
Hi Eric,

> Basic questions.
> 1) Do you have panic on oops set?
>    I think that setting more than anything else will be the
>    difference in what shows up in /var/log/messages.

Yes, in my case panic_on_oops was set with 1

> 2) Have you tried a dry run and confirmed you can get a crash dump?

No, I haven't tried this.

> 3) Are you using kvm
kvm? kernel virtual memory?

thanks,
Seetharam

> 
> 
> Eric
> 
> 
>> Thanks and Regards
>> 
>>  Seetharam
>> 
>> 
>>> -----Original Message-----
>>> From: nhorman@redhat.com
>>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
>>> To: dseetharam@inbox.com
>>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>>> panic/oops/crash
>>> 
>>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
>>>> Hi,
>>>> 
>>>> I have encounter a problem called /var/log/messages does not contain
>>>> any
>>>> information
>>>> on kernel crash dump.
>>>> 
>>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
>>>> /var/log/messages.
>>>> After enabling KEXEC feature those info was missed.
>>>> 
>>>> How can we get those missed info in /var/log/messages even when KEXEC
>>>> feature is enabled.
>>>> 
>>>> Any help will appreciate.
>>>> 
>>> run crash on the resultant vmcore file that kexec can produce/save and
>>> get the
>>> dmesg log out of it.
>>> alternatively you can use makedumpfile with the --dump-dmesg log
>>> 
>>> Neil
>>> 
>>>> 
>>>> Thanks and Regards
>>>> Seetharam
>>>> 
>>>> ____________________________________________________________
>>>> Receive Notifications of Incoming Messages
>>>> Easily monitor multiple email accounts & access them with a click.
>>>> Visit http://www.inbox.com/notifier and check it out!
>>>> 
>>>> _______________________________________________
>>>> kexec mailing list
>>>> kexec@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/kexec
>> 
>> ____________________________________________________________
>> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
>> http://www.inbox.com/smileys
>> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
>> and most webmails

____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Eric W. Biederman]

Dharmosoth Seetharam <dseetharam@inbox.com> writes:

>> -----Original Message-----
>> From: ebiederm@xmission.com
>> Sent: Sat, 13 Jun 2009 00:11:25 -0700
>> To: dseetharam@inbox.com
>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>> panic/oops/crash
>> 
>> Dharmosoth Seetharam <dseetharam@inbox.com> writes:
>> 
>>> Hi Neil,
>>> 
>>> Thanks for your quick reply.
>>> 
>>> What you have suggested is nice when we got/save crash dump (i.e vmcore)
>>> What if we failed to save/capture vmcore info or vmcore is corrupted
>>> etc. then we dont get any information
>>> regarding the panic/crash/oops at anywhere?
>> 
> Hi Eric,
>
>> Basic questions.
>> 1) Do you have panic on oops set?
>>    I think that setting more than anything else will be the
>>    difference in what shows up in /var/log/messages.
>
> Yes, in my case panic_on_oops was set with 1

That is the reason thing don't show up in /var/log/messages
because you panic before syslog has a chance to write the messages.

>> 2) Have you tried a dry run and confirmed you can get a crash dump?
>
> No, I haven't tried this.

It sounds like you have not been getting the coredumps when problems
happen.  Running a simple test run to  Alt-sysrq-c to confirm
that things are basically setup ok is a good idea.

>> 3) Are you using kvm
> kvm? kernel virtual memory?

Kernel virtual machine.  In particular.
Hardware acceleration for virtual machines.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

> -----Original Message-----
> From: ebiederm@xmission.com
> Sent: Sat, 13 Jun 2009 05:51:25 -0700
> To: dseetharam@inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
> 
> Dharmosoth Seetharam <dseetharam@inbox.com> writes:
> 
>>> -----Original Message-----
>>> From: ebiederm@xmission.com
>>> Sent: Sat, 13 Jun 2009 00:11:25 -0700
>>> To: dseetharam@inbox.com
>>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>>> panic/oops/crash
>>> 
>>> Dharmosoth Seetharam <dseetharam@inbox.com> writes:
>>> 
>>>> Hi Neil,
>>>> 
>>>> Thanks for your quick reply.
>>>> 
>>>> What you have suggested is nice when we got/save crash dump (i.e
>>>> vmcore)
>>>> What if we failed to save/capture vmcore info or vmcore is corrupted
>>>> etc. then we dont get any information
>>>> regarding the panic/crash/oops at anywhere?
>>> 
>> Hi Eric,
>> 
>>> Basic questions.
>>> 1) Do you have panic on oops set?
>>>    I think that setting more than anything else will be the
>>>    difference in what shows up in /var/log/messages.
>> 
>> Yes, in my case panic_on_oops was set with 1
> 
> That is the reason thing don't show up in /var/log/messages
> because you panic before syslog has a chance to write the messages.

Thanks.
So, if panic_on_oops set with 0(default), we will get all the 
needed info like stack traces, bug info etc. in /var/log/messages then reboots.

Or do we need to apply any specific patches to get info in /var/log/messages?

> 
>>> 2) Have you tried a dry run and confirmed you can get a crash dump?
>> 
>> No, I haven't tried this.
> 
> It sounds like you have not been getting the coredumps when problems
> happen.  Running a simple test run to  Alt-sysrq-c to confirm
> that things are basically setup ok is a good idea.
> 
Sorry, I misunderstood your question.
I have gave dry run and confirmed that the dumps are getting saved in particular dir and able to analyze.

I did in both the ways like
1 - echo c > /proc/sysrq-trigger
2 -Alt -sysrq -c



thanks,
Seetharam

>>> 3) Are you using kvm
>> kvm? kernel virtual memory?
> 
> Kernel virtual machine.  In particular.
> Hardware acceleration for virtual machines.
> 
> Eric

____________________________________________________________
FREE ONLINE PHOTOSHARING - Share your photos online with your friends and family!
Visit http://www.inbox.com/photosharing to find out more!

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Eric W. Biederman]

Dharmosoth Seetharam <dseetharam@inbox.com> writes:

>>>> Basic questions.
>>>> 1) Do you have panic on oops set?
>>>>    I think that setting more than anything else will be the
>>>>    difference in what shows up in /var/log/messages.
>>> 
>>> Yes, in my case panic_on_oops was set with 1
>> 
>> That is the reason thing don't show up in /var/log/messages
>> because you panic before syslog has a chance to write the messages.
>
> Thanks.
> So, if panic_on_oops set with 0(default), we will get all the 
> needed info like stack traces, bug info etc. in /var/log/messages then reboots.
>
> Or do we need to apply any specific patches to get info in /var/log/messages?
>
>> 
>>>> 2) Have you tried a dry run and confirmed you can get a crash dump?
>>> 
>>> No, I haven't tried this.
>> 
>> It sounds like you have not been getting the coredumps when problems
>> happen.  Running a simple test run to  Alt-sysrq-c to confirm
>> that things are basically setup ok is a good idea.
>> 
> Sorry, I misunderstood your question.
> I have gave dry run and confirmed that the dumps are getting saved in particular dir and able to analyze.
>
> I did in both the ways like
> 1 - echo c > /proc/sysrq-trigger
> 2 -Alt -sysrq -c

Then unless you are having problems capturing core dumps in real
failure situations it sounds like all is well with the world.

A kernel oops is normally not fatal and the kernel tries to limp along
allowing for better diagnostics etc.  This allows klogd to read the
kernels message buffer and pass the kernel messages to syslog which
writes the messages to /var/log/messages.

After a kernel panic nothing is allowed to happen which prevents
user space from writing to /var/log/messages in the usual way.

If you want data in /var/log/messages it appears you have two choices.
1) Post process a core dump as Neil suggested and feed the kernel log
   buffer to syslog.
2) disable panic on oops.  The system will continue to limp along allowing
   user space to write to /var/log/messages.

Eric






_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

> -----Original Message-----
> From: ebiederm@xmission.com
> Sent: Sat, 13 Jun 2009 07:08:50 -0700
> To: dseetharam@inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
> 
> Dharmosoth Seetharam <dseetharam@inbox.com> writes:
> 
>>>>> Basic questions.
>>>>> 1) Do you have panic on oops set?
>>>>>    I think that setting more than anything else will be the
>>>>>    difference in what shows up in /var/log/messages.
>>>> 
>>>> Yes, in my case panic_on_oops was set with 1
>>> 
>>> That is the reason thing don't show up in /var/log/messages
>>> because you panic before syslog has a chance to write the messages.
>> 
>> Thanks.
>> So, if panic_on_oops set with 0(default), we will get all the
>> needed info like stack traces, bug info etc. in /var/log/messages then
>> reboots.
>> 
>> Or do we need to apply any specific patches to get info in
>> /var/log/messages?
>> 
>>> 
>>>>> 2) Have you tried a dry run and confirmed you can get a crash dump?
>>>> 
>>>> No, I haven't tried this.
>>> 
>>> It sounds like you have not been getting the coredumps when problems
>>> happen.  Running a simple test run to  Alt-sysrq-c to confirm
>>> that things are basically setup ok is a good idea.
>>> 
>> Sorry, I misunderstood your question.
>> I have gave dry run and confirmed that the dumps are getting saved in
>> particular dir and able to analyze.
>> 
>> I did in both the ways like
>> 1 - echo c > /proc/sysrq-trigger
>> 2 -Alt -sysrq -c
> 
> Then unless you are having problems capturing core dumps in real
> failure situations it sounds like all is well with the world.
> 
> A kernel oops is normally not fatal and the kernel tries to limp along
> allowing for better diagnostics etc.  This allows klogd to read the
> kernels message buffer and pass the kernel messages to syslog which
> writes the messages to /var/log/messages.
> 
> After a kernel panic nothing is allowed to happen which prevents
> user space from writing to /var/log/messages in the usual way.
> 
> If you want data in /var/log/messages it appears you have two choices.
> 1) Post process a core dump as Neil suggested and feed the kernel log
>    buffer to syslog.
> 2) disable panic on oops.  The system will continue to limp along
> allowing
>    user space to write to /var/log/messages.
> 
> Eric

Thanks Eric.

I will try all the ways as you suggested and let you know the outcome.

regards,
Seetharam

____________________________________________________________
FREE ONLINE PHOTOSHARING - Share your photos online with your friends and family!
Visit http://www.inbox.com/photosharing to find out more!

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Neil Horman]

On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
> 
> Hi Neil,
> 
> Thanks for your quick reply.
> 
> What you have suggested is nice when we got/save crash dump (i.e vmcore)
> What if we failed to save/capture vmcore info or vmcore is corrupted etc. then we dont get any information
> regarding the panic/crash/oops at anywhere?
> 
If the vmcore is corrupted to the point where makedumpfile can't extract the
dmesg log, then its certain that syslog wouldn't have been able to write
anything to /var/log/messages either.  As a last resort you can try run strings
on the corrupted vmcore and see if the dmesg log is still in there somewhere.

If you forgot to capture the vmcore, theres not much more help I can give you,
Im afraid.  You'll just have to make sure you capture it next time.  If you only
need the backtrace, and don't want the rest of the vmcore, you can make kdump
just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt

Neil

>  
> 
> 
> Thanks and Regards
> 
>  Seetharam
> 
> 
> > -----Original Message-----
> > From: nhorman@redhat.com
> > Sent: Fri, 12 Jun 2009 16:05:13 -0400
> > To: dseetharam@inbox.com
> > Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> > panic/oops/crash
> > 
> > On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
> >> Hi,
> >> 
> >> I have encounter a problem called /var/log/messages does not contain any
> >> information
> >> on kernel crash dump.
> >> 
> >> Prior to enabling KEXEC we saw stack traces and bug info etc in
> >> /var/log/messages.
> >> After enabling KEXEC feature those info was missed.
> >> 
> >> How can we get those missed info in /var/log/messages even when KEXEC
> >> feature is enabled.
> >> 
> >> Any help will appreciate.
> >> 
> > run crash on the resultant vmcore file that kexec can produce/save and
> > get the
> > dmesg log out of it.
> > alternatively you can use makedumpfile with the --dump-dmesg log
> > 
> > Neil
> > 
> >> 
> >> Thanks and Regards
> >> Seetharam
> >> 
> >> ____________________________________________________________
> >> Receive Notifications of Incoming Messages
> >> Easily monitor multiple email accounts & access them with a click.
> >> Visit http://www.inbox.com/notifier and check it out!
> >> 
> >> _______________________________________________
> >> kexec mailing list
> >> kexec@lists.infradead.org
> >> http://lists.infradead.org/mailman/listinfo/kexec
> 
> ____________________________________________________________
> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at http://www.inbox.com/smileys
> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™ and most webmails

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

Hi,

I have experimented few things and results are below.

Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
        When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
        and then continue(not switching to dump-saving kernel).

        What I am looking is - I want both the things to be happen.
        that is - it should log the appropriate messages in /var/log/messsages and
        should be able to take kernel dump.

case 2) In general, When panic has occurred we won't log any type of messages in
        /var/log/messages ... am I right ?
        If KEXEC is enabled and dump-saving kernel is loaded then 
        will switch to it and save the kernel dump.

        I have added the dump_stack() function call in the panic() function 
        before calling crash_kexec with few seconds of delay. It is behaving 
        like randomly (ie. some time stack traces are logged in /var/log/messages and 
        not for rest of the time)

        What I am looking is - I want to log the  appropriate messages in /var/log/messages
        and then switch to dump-saving kernel.

case 3) In general, When die/nmi_die called we used to log the appropriate messages in /var/log/message
        and then saving kernel dump will be depends on KEXEC enable and loading dump-saving kernel.
        --- am I right ?
        
        I have added the delay in die() function with few seconds before calling crash_kexec()
        But I never seen messages logged in /var/log/messages, I can see those on serial console.

        Here also my intention is want both, ie. log the appropriate messages in /var/log/messages
        and then take the action as per KEXEC enabled.



Please advice me, What we can do to achieve those things.

Thanks for helping me here.


regards,
Seetharam





> -----Original Message-----
> From: nhorman@redhat.com
> Sent: Sat, 13 Jun 2009 14:07:36 -0400
> To: dseetharam@inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
> 
> On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
>> 
>> Hi Neil,
>> 
>> Thanks for your quick reply.
>> 
>> What you have suggested is nice when we got/save crash dump (i.e vmcore)
>> What if we failed to save/capture vmcore info or vmcore is corrupted
>> etc. then we dont get any information
>> regarding the panic/crash/oops at anywhere?
>> 
> If the vmcore is corrupted to the point where makedumpfile can't extract
> the
> dmesg log, then its certain that syslog wouldn't have been able to write
> anything to /var/log/messages either.  As a last resort you can try run
> strings
> on the corrupted vmcore and see if the dmesg log is still in there
> somewhere.
> 
> If you forgot to capture the vmcore, theres not much more help I can give
> you,
> Im afraid.  You'll just have to make sure you capture it next time.  If
> you only
> need the backtrace, and don't want the rest of the vmcore, you can make
> kdump
> just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
> 
> Neil
> 


>> 
>> 
>> 
>> Thanks and Regards
>> 
>>  Seetharam
>> 
>> 
>>> -----Original Message-----
>>> From: nhorman@redhat.com
>>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
>>> To: dseetharam@inbox.com
>>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>>> panic/oops/crash
>>> 
>>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
>>>> Hi,
>>>> 
>>>> I have encounter a problem called /var/log/messages does not contain
>>>> any
>>>> information
>>>> on kernel crash dump.
>>>> 
>>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
>>>> /var/log/messages.
>>>> After enabling KEXEC feature those info was missed.
>>>> 
>>>> How can we get those missed info in /var/log/messages even when KEXEC
>>>> feature is enabled.
>>>> 
>>>> Any help will appreciate.
>>>> 
>>> run crash on the resultant vmcore file that kexec can produce/save and
>>> get the
>>> dmesg log out of it.
>>> alternatively you can use makedumpfile with the --dump-dmesg log
>>> 
>>> Neil
>>> 
>>>> 
>>>> Thanks and Regards
>>>> Seetharam
>>>> 
>>>> ____________________________________________________________
>>>> Receive Notifications of Incoming Messages
>>>> Easily monitor multiple email accounts & access them with a click.
>>>> Visit http://www.inbox.com/notifier and check it out!
>>>> 
>>>> _______________________________________________
>>>> kexec mailing list
>>>> kexec@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/kexec
>> 
>> ____________________________________________________________
>> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
>> http://www.inbox.com/smileys
>> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
>> and most webmails

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [Fastboot] /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Haren Myneni]

[-- Attachment #1.1: Type: text/plain, Size: 5853 bytes --]



fastboot-bounces@lists.linux-foundation.org wrote on 06/15/2009 09:43:02
PM:

> Dharmosoth Seetharam <dseetharam@inbox.com>
>
> Hi,
>
> I have experimented few things and results are below.
>
> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>         When BUG/oops occurred it is able to log the diagnostics to
> /var/log/messages
>         and then continue(not switching to dump-saving kernel).
>
>         What I am looking is - I want both the things to be happen.
>         that is - it should log the appropriate messages in /var/
> log/messsages and
>         should be able to take kernel dump.

when panic_on_oops is not set, it does not call crash_kexec() - change in
kexec_should_crash() in kernel/kexec.c

>
> case 2) In general, When panic has occurred we won't log any type of
> messages in
>         /var/log/messages ... am I right ?
>         If KEXEC is enabled and dump-saving kernel is loaded then
>         will switch to it and save the kernel dump.
>
>         I have added the dump_stack() function call in the panic()
function
>         before calling crash_kexec with few seconds of delay. It is
behaving
>         like randomly (ie. some time stack traces are logged in /
> var/log/messages and
>         not for rest of the time)
>
>         What I am looking is - I want to log the  appropriate
> messages in /var/log/messages
>         and then switch to dump-saving kernel.
>

printk messages in /var/log/messages is depends on how fast klogd flushes
printk buffer.

> case 3) In general, When die/nmi_die called we used to log the
> appropriate messages in /var/log/message
>         and then saving kernel dump will be depends on KEXEC enable
> and loading dump-saving kernel.
>         --- am I right ?
>
>         I have added the delay in die() function with few seconds
> before calling crash_kexec()
>         But I never seen messages logged in /var/log/messages, I can
> see those on serial console.
>
>         Here also my intention is want both, ie. log the appropriate
> messages in /var/log/messages
>         and then take the action as per KEXEC enabled.

printk write to the console and its buffer, As I said, depends on time for
klogd flushes printk buffer.


>
>
>
> Please advice me, What we can do to achieve those things.
>
> Thanks for helping me here.
>
>
> regards,
> Seetharam
>
>
>
>
>
> > -----Original Message-----
> > From: nhorman@redhat.com
> > Sent: Sat, 13 Jun 2009 14:07:36 -0400
> > To: dseetharam@inbox.com
> > Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> > panic/oops/crash
> >
> > On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
> >>
> >> Hi Neil,
> >>
> >> Thanks for your quick reply.
> >>
> >> What you have suggested is nice when we got/save crash dump (i.e
vmcore)
> >> What if we failed to save/capture vmcore info or vmcore is corrupted
> >> etc. then we dont get any information
> >> regarding the panic/crash/oops at anywhere?
> >>
> > If the vmcore is corrupted to the point where makedumpfile can't
extract
> > the
> > dmesg log, then its certain that syslog wouldn't have been able to
write
> > anything to /var/log/messages either.  As a last resort you can try run
> > strings
> > on the corrupted vmcore and see if the dmesg log is still in there
> > somewhere.
> >
> > If you forgot to capture the vmcore, theres not much more help I can
give
> > you,
> > Im afraid.  You'll just have to make sure you capture it next time.  If
> > you only
> > need the backtrace, and don't want the rest of the vmcore, you can make
> > kdump
> > just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
> >
> > Neil
> >
>
>
> >>
> >>
> >>
> >> Thanks and Regards
> >>
> >>  Seetharam
> >>
> >>
> >>> -----Original Message-----
> >>> From: nhorman@redhat.com
> >>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
> >>> To: dseetharam@inbox.com
> >>> Subject: Re: /var/log/messages doesn't have crash info when kernel
gets
> >>> panic/oops/crash
> >>>
> >>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
> >>>> Hi,
> >>>>
> >>>> I have encounter a problem called /var/log/messages does not contain
> >>>> any
> >>>> information
> >>>> on kernel crash dump.
> >>>>
> >>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
> >>>> /var/log/messages.
> >>>> After enabling KEXEC feature those info was missed.
> >>>>
> >>>> How can we get those missed info in /var/log/messages even when
KEXEC
> >>>> feature is enabled.
> >>>>
> >>>> Any help will appreciate.
> >>>>
> >>> run crash on the resultant vmcore file that kexec can produce/save
and
> >>> get the
> >>> dmesg log out of it.
> >>> alternatively you can use makedumpfile with the --dump-dmesg log
> >>>
> >>> Neil
> >>>
> >>>>
> >>>> Thanks and Regards
> >>>> Seetharam
> >>>>
> >>>> ____________________________________________________________
> >>>> Receive Notifications of Incoming Messages
> >>>> Easily monitor multiple email accounts & access them with a click.
> >>>> Visit http://www.inbox.com/notifier and check it out!
> >>>>
> >>>> _______________________________________________
> >>>> kexec mailing list
> >>>> kexec@lists.infradead.org
> >>>> http://lists.infradead.org/mailman/listinfo/kexec
> >>
> >> ____________________________________________________________
> >> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
> >> http://www.inbox.com/smileys
> >> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
> >> and most webmails
> _______________________________________________
> fastboot mailing list
> fastboot@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/fastboot

[-- Attachment #1.2: Type: text/html, Size: 8570 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Eric W. Biederman]

Dharmosoth Seetharam <dseetharam@inbox.com> writes:

> Hi,
>
> I have experimented few things and results are below.
>
> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>         When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
>         and then continue(not switching to dump-saving kernel).
>
>         What I am looking is - I want both the things to be happen.
>         that is - it should log the appropriate messages in /var/log/messsages and
>         should be able to take kernel dump.

You can't have it.  Sorry.
Keeping user space alive is something that simply can not be done reliably.

> Please advice me, What we can do to achieve those things.

The only scenario I see possible is to take the dump.  Use
makedumpfile or the crash log command to extract the kernel messages
and put them in the /var/log/messages.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Eric W. Biederman]

Dharmosoth Seetharam <dseetharam@inbox.com> writes:

> Hi,
>
> I have experimented few things and results are below.
>
> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>         When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
>         and then continue(not switching to dump-saving kernel).
>
>         What I am looking is - I want both the things to be happen.
>         that is - it should log the appropriate messages in /var/log/messsages and
>         should be able to take kernel dump.

You can't have it.  Sorry.
Keeping user space alive is something that simply can not be done reliably.

> Please advice me, What we can do to achieve those things.

The only scenario I see possible is to take the dump.  Use
makedumpfile or the crash log command to extract the kernel messages
and put them in the /var/log/messages.

Eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Neil Horman]

On Mon, Jun 15, 2009 at 08:43:02PM -0800, Dharmosoth Seetharam wrote:
> Hi,
> 
> I have experimented few things and results are below.
> 
> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>         When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
>         and then continue(not switching to dump-saving kernel).
> 
>         What I am looking is - I want both the things to be happen.
>         that is - it should log the appropriate messages in /var/log/messsages and
>         should be able to take kernel dump.
> 
> case 2) In general, When panic has occurred we won't log any type of messages in
>         /var/log/messages ... am I right ?
>         If KEXEC is enabled and dump-saving kernel is loaded then 
>         will switch to it and save the kernel dump.
> 
>         I have added the dump_stack() function call in the panic() function 
>         before calling crash_kexec with few seconds of delay. It is behaving 
>         like randomly (ie. some time stack traces are logged in /var/log/messages and 
>         not for rest of the time)
> 
>         What I am looking is - I want to log the  appropriate messages in /var/log/messages
>         and then switch to dump-saving kernel.
> 
Then what you want to do is, enable kdump, and set your kernel up to
panic_on_oops.  Then, configure kdump to extract the dmesg log from
/proc/vmcore, and write that to a file (either a file you create, or to the end
of /var/log/messages).  Most distributions kdump configurations allow for
something simmilar to this.  The new config method I've got in rawhide should
make this very easy.

The problem is that when you panic the box (which is what has to happen for
kdump to start), the syslog daemon doesn't get a chance to run first, so theres
no way to write the oops backtrace to var/log/messages.  You'll need to tell
kdump to do that for you after the fact.

> case 3) In general, When die/nmi_die called we used to log the appropriate messages in /var/log/message
>         and then saving kernel dump will be depends on KEXEC enable and loading dump-saving kernel.
>         --- am I right ?
>         
>         I have added the delay in die() function with few seconds before calling crash_kexec()
>         But I never seen messages logged in /var/log/messages, I can see those on serial console.
> 
See above, when we printk those messages, klogd/syslogd needs to run to get them
from the console buffer and write them to /var/log/messages.  Theres no
guarantee that will ever happen.

>         Here also my intention is want both, ie. log the appropriate messages in /var/log/messages
>         and then take the action as per KEXEC enabled.
> 
> 
> 
> Please advice me, What we can do to achieve those things.
> 
See above, use makedumpfile to extract the dmesg log from /proc/vmcore in kdump,
then use a custom script to append the log to /var/log/messages.  In fedora that
will be equivalent to using the core_collector and kdump_post options in
/etc/kdump.conf.

Neil

> Thanks for helping me here.
> 
> 
> regards,
> Seetharam
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: nhorman@redhat.com
> > Sent: Sat, 13 Jun 2009 14:07:36 -0400
> > To: dseetharam@inbox.com
> > Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> > panic/oops/crash
> > 
> > On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
> >> 
> >> Hi Neil,
> >> 
> >> Thanks for your quick reply.
> >> 
> >> What you have suggested is nice when we got/save crash dump (i.e vmcore)
> >> What if we failed to save/capture vmcore info or vmcore is corrupted
> >> etc. then we dont get any information
> >> regarding the panic/crash/oops at anywhere?
> >> 
> > If the vmcore is corrupted to the point where makedumpfile can't extract
> > the
> > dmesg log, then its certain that syslog wouldn't have been able to write
> > anything to /var/log/messages either.  As a last resort you can try run
> > strings
> > on the corrupted vmcore and see if the dmesg log is still in there
> > somewhere.
> > 
> > If you forgot to capture the vmcore, theres not much more help I can give
> > you,
> > Im afraid.  You'll just have to make sure you capture it next time.  If
> > you only
> > need the backtrace, and don't want the rest of the vmcore, you can make
> > kdump
> > just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
> > 
> > Neil
> > 
> 
> 
> >> 
> >> 
> >> 
> >> Thanks and Regards
> >> 
> >>  Seetharam
> >> 
> >> 
> >>> -----Original Message-----
> >>> From: nhorman@redhat.com
> >>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
> >>> To: dseetharam@inbox.com
> >>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> >>> panic/oops/crash
> >>> 
> >>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
> >>>> Hi,
> >>>> 
> >>>> I have encounter a problem called /var/log/messages does not contain
> >>>> any
> >>>> information
> >>>> on kernel crash dump.
> >>>> 
> >>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
> >>>> /var/log/messages.
> >>>> After enabling KEXEC feature those info was missed.
> >>>> 
> >>>> How can we get those missed info in /var/log/messages even when KEXEC
> >>>> feature is enabled.
> >>>> 
> >>>> Any help will appreciate.
> >>>> 
> >>> run crash on the resultant vmcore file that kexec can produce/save and
> >>> get the
> >>> dmesg log out of it.
> >>> alternatively you can use makedumpfile with the --dump-dmesg log
> >>> 
> >>> Neil
> >>> 
> >>>> 
> >>>> Thanks and Regards
> >>>> Seetharam
> >>>> 
> >>>> ____________________________________________________________
> >>>> Receive Notifications of Incoming Messages
> >>>> Easily monitor multiple email accounts & access them with a click.
> >>>> Visit http://www.inbox.com/notifier and check it out!
> >>>> 
> >>>> _______________________________________________
> >>>> kexec mailing list
> >>>> kexec@lists.infradead.org
> >>>> http://lists.infradead.org/mailman/listinfo/kexec
> >> 
> >> ____________________________________________________________
> >> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
> >> http://www.inbox.com/smileys
> >> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
> >> and most webmails

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam]

Thanks a lot.
Seetharam

> -----Original Message-----
> From: nhorman@redhat.com
> Sent: Tue, 16 Jun 2009 07:24:25 -0400
> To: dseetharam@inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
> 
> On Mon, Jun 15, 2009 at 08:43:02PM -0800, Dharmosoth Seetharam wrote:
>> Hi,
>> 
>> I have experimented few things and results are below.
>> 
>> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>>         When BUG/oops occurred it is able to log the diagnostics to
>> /var/log/messages
>>         and then continue(not switching to dump-saving kernel).
>> 
>>         What I am looking is - I want both the things to be happen.
>>         that is - it should log the appropriate messages in
>> /var/log/messsages and
>>         should be able to take kernel dump.
>> 
>> case 2) In general, When panic has occurred we won't log any type of
>> messages in
>>         /var/log/messages ... am I right ?
>>         If KEXEC is enabled and dump-saving kernel is loaded then
>>         will switch to it and save the kernel dump.
>> 
>>         I have added the dump_stack() function call in the panic()
>> function
>>         before calling crash_kexec with few seconds of delay. It is
>> behaving
>>         like randomly (ie. some time stack traces are logged in
>> /var/log/messages and
>>         not for rest of the time)
>> 
>>         What I am looking is - I want to log the  appropriate messages
>> in /var/log/messages
>>         and then switch to dump-saving kernel.
>> 
> Then what you want to do is, enable kdump, and set your kernel up to
> panic_on_oops.  Then, configure kdump to extract the dmesg log from
> /proc/vmcore, and write that to a file (either a file you create, or to
> the end
> of /var/log/messages).  Most distributions kdump configurations allow for
> something simmilar to this.  The new config method I've got in rawhide
> should
> make this very easy.
> 
> The problem is that when you panic the box (which is what has to happen
> for
> kdump to start), the syslog daemon doesn't get a chance to run first, so
> theres
> no way to write the oops backtrace to var/log/messages.  You'll need to
> tell
> kdump to do that for you after the fact.
> 
>> case 3) In general, When die/nmi_die called we used to log the
>> appropriate messages in /var/log/message
>>         and then saving kernel dump will be depends on KEXEC enable and
>> loading dump-saving kernel.
>>         --- am I right ?
>> 
>>         I have added the delay in die() function with few seconds before
>> calling crash_kexec()
>>         But I never seen messages logged in /var/log/messages, I can see
>> those on serial console.
>> 
> See above, when we printk those messages, klogd/syslogd needs to run to
> get them
> from the console buffer and write them to /var/log/messages.  Theres no
> guarantee that will ever happen.
> 
>>         Here also my intention is want both, ie. log the appropriate
>> messages in /var/log/messages
>>         and then take the action as per KEXEC enabled.
>> 
>> 
>> 
>> Please advice me, What we can do to achieve those things.
>> 
> See above, use makedumpfile to extract the dmesg log from /proc/vmcore in
> kdump,
> then use a custom script to append the log to /var/log/messages.  In
> fedora that
> will be equivalent to using the core_collector and kdump_post options in
> /etc/kdump.conf.
> 
> Neil
> 
>> Thanks for helping me here.
>> 
>> 
>> regards,
>> Seetharam
>> 
>> 
>> 
>> 
>> 
>>> -----Original Message-----
>>> From: nhorman@redhat.com
>>> Sent: Sat, 13 Jun 2009 14:07:36 -0400
>>> To: dseetharam@inbox.com
>>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>>> panic/oops/crash
>>> 
>>> On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
>>>> 
>>>> Hi Neil,
>>>> 
>>>> Thanks for your quick reply.
>>>> 
>>>> What you have suggested is nice when we got/save crash dump (i.e
>>>> vmcore)
>>>> What if we failed to save/capture vmcore info or vmcore is corrupted
>>>> etc. then we dont get any information
>>>> regarding the panic/crash/oops at anywhere?
>>>> 
>>> If the vmcore is corrupted to the point where makedumpfile can't
>>> extract
>>> the
>>> dmesg log, then its certain that syslog wouldn't have been able to
>>> write
>>> anything to /var/log/messages either.  As a last resort you can try run
>>> strings
>>> on the corrupted vmcore and see if the dmesg log is still in there
>>> somewhere.
>>> 
>>> If you forgot to capture the vmcore, theres not much more help I can
>>> give
>>> you,
>>> Im afraid.  You'll just have to make sure you capture it next time.  If
>>> you only
>>> need the backtrace, and don't want the rest of the vmcore, you can make
>>> kdump
>>> just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
>>> 
>>> Neil
>>> 
>> 
>> 
>>>> 
>>>> 
>>>> 
>>>> Thanks and Regards
>>>> 
>>>>  Seetharam
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: nhorman@redhat.com
>>>>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
>>>>> To: dseetharam@inbox.com
>>>>> Subject: Re: /var/log/messages doesn't have crash info when kernel
>>>>> gets
>>>>> panic/oops/crash
>>>>> 
>>>>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I have encounter a problem called /var/log/messages does not contain
>>>>>> any
>>>>>> information
>>>>>> on kernel crash dump.
>>>>>> 
>>>>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
>>>>>> /var/log/messages.
>>>>>> After enabling KEXEC feature those info was missed.
>>>>>> 
>>>>>> How can we get those missed info in /var/log/messages even when
>>>>>> KEXEC
>>>>>> feature is enabled.
>>>>>> 
>>>>>> Any help will appreciate.
>>>>>> 
>>>>> run crash on the resultant vmcore file that kexec can produce/save
>>>>> and
>>>>> get the
>>>>> dmesg log out of it.
>>>>> alternatively you can use makedumpfile with the --dump-dmesg log
>>>>> 
>>>>> Neil
>>>>> 
>>>>>> 
>>>>>> Thanks and Regards
>>>>>> Seetharam
>>>>>> 
>>>>>> ____________________________________________________________
>>>>>> Receive Notifications of Incoming Messages
>>>>>> Easily monitor multiple email accounts & access them with a click.
>>>>>> Visit http://www.inbox.com/notifier and check it out!
>>>>>> 
>>>>>> _______________________________________________
>>>>>> kexec mailing list
>>>>>> kexec@lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/kexec
>>>> 
>>>> ____________________________________________________________
>>>> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
>>>> http://www.inbox.com/smileys
>>>> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
>>>> and most webmails

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec