Plea for help

Plea for help

[Ian Pilcher]

I posted a couple of weeks ago about a kernel panic that shows up when I
boot my Abit VP6 (dual-PIII) with ACPI turned on.  It turns out that the
kernel boots just fine with ACPI on when I build it with
CONFIG_DEBUG_SLAB=y.

Alan Cox suggested that this is probably due to code that kmallocs
memory and then relies on its content without clearing it, but I never
received any response to my request for suggestions on how to identify
the offending code.

I've just spent several hours adding printk statements to the ACPI code,
and I've reached a point where the oops message is overwriting portions
of my printk messages.  I have the sinking feeling that this indicates
that the process executing the ACPI code is not the one that's oops'ing.

I'm stuck.  How can a relative newbie debug a problem like this, or at
least gather enough information to enable someone with more expertise to
do so?

TIA.
-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Micha Feigin]

On Thu, Dec 18, 2003 at 11:41:13PM -0600, Ian Pilcher wrote:
> I posted a couple of weeks ago about a kernel panic that shows up when I
> boot my Abit VP6 (dual-PIII) with ACPI turned on.  It turns out that the
> kernel boots just fine with ACPI on when I build it with
> CONFIG_DEBUG_SLAB=y.
> 
> Alan Cox suggested that this is probably due to code that kmallocs
> memory and then relies on its content without clearing it, but I never
> received any response to my request for suggestions on how to identify
> the offending code.
> 
> I've just spent several hours adding printk statements to the ACPI code,
> and I've reached a point where the oops message is overwriting portions
> of my printk messages.  I have the sinking feeling that this indicates
> that the process executing the ACPI code is not the one that's oops'ing.
> 
> I'm stuck.  How can a relative newbie debug a problem like this, or at
> least gather enough information to enable someone with more expertise to
> do so?
> 

If you say that CONFIG_DEBUG_SLAB=y solves the problem, you could start
by grepping in the kernel for CONFIG_DEBUG_SLAB and see what its doing.
This sounds like one of two things.
Either CONFIG_DEBUG_SLAB=y clears some memory which isn't cleared
properly somewhere else (there are quite a few kernel structures that
are allocated as slabs iirc), or its a timing problem that
CONFIG_DEBUG_SLAB solves by adding a printk somewhere that causes a
short delay (the second one will probably be a lot harder to solve).
I'll see if I have some time to look at it later.

> TIA.
> -- 
> ========================================================================
> Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
> ========================================================================
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Acpi-devel mailing list
> Acpi-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/acpi-devel
> 


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Ian Pilcher]

Brian Perkins wrote:
> First of all, what's the panic like?
> Does it give a function?
> You do have all of the kernel hacking options on, don't you?
> 

You can see the panic at

   http://home.comcast.net/~i.pilcher/panics/1/panic.txt

The config used to produce it is at

   http://home.comcast.net/~i.pilcher/panics/1/config

Unfortunately, if I turn CONFIG_FRAME_POINTER on I don't get the panic,
just the hang at

   http://home.comcast.net/~i.pilcher/panics/2/hang.txt

Config at

   http://home.comcast.net/~i.pilcher/panics/2/config

And as I previously mentioned, turning CONFIG_DEBUG_SLAB on prevents
the panic entirely.

-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Randy.Dunlap]

On Fri, 19 Dec 2003 18:05:05 -0600 Ian Pilcher <i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org> wrote:

| Brian Perkins wrote:
| > First of all, what's the panic like?
| > Does it give a function?
| > You do have all of the kernel hacking options on, don't you?
| > 
| 
| You can see the panic at
| 
|    http://home.comcast.net/~i.pilcher/panics/1/panic.txt
| 
| The config used to produce it is at
| 
|    http://home.comcast.net/~i.pilcher/panics/1/config
| 
| Unfortunately, if I turn CONFIG_FRAME_POINTER on I don't get the panic,
| just the hang at
| 
|    http://home.comcast.net/~i.pilcher/panics/2/hang.txt
| 
| Config at
| 
|    http://home.comcast.net/~i.pilcher/panics/2/config
| 
| And as I previously mentioned, turning CONFIG_DEBUG_SLAB on prevents
| the panic entirely.

Can you also put the System.map file for that kernel on your website
so that we can correlate kernel memory addresses?

--
~Randy
MOTD:  Always include version info.


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Ian Pilcher]

Randy.Dunlap wrote:

> Can you also put the System.map file for that kernel on your website
> so that we can correlate kernel memory addresses?

They're up now.  The System.map for the panic (CONFIG_FRAME_POINTER
off) is at

   http://home.comcast.net/~i.pilcher/panics/1/System.map.txt

The map for the hang (CONFIG_FRAME_POINTER on) is at

   http://home.comcast.net/~i.pilcher/panics/2/System.map.txt

-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Randy.Dunlap]

On Fri, 19 Dec 2003 19:09:22 -0600 Ian Pilcher <i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org> wrote:

| Randy.Dunlap wrote:
| 
| > Can you also put the System.map file for that kernel on your website
| > so that we can correlate kernel memory addresses?
| 
| They're up now.  The System.map for the panic (CONFIG_FRAME_POINTER
| off) is at
| 
|    http://home.comcast.net/~i.pilcher/panics/1/System.map.txt
| 
| The map for the hang (CONFIG_FRAME_POINTER on) is at
| 
|    http://home.comcast.net/~i.pilcher/panics/2/System.map.txt


and exactly what kernel version is this?

--
~Randy


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Ian Pilcher]

Randy.Dunlap wrote:
> 
> and exactly what kernel version is this?
> 

2.4.23 with the 20031203 ACPI patch.  I get basically the same behavior
with unpatched 2.4.23 or any of the Fedora Core kernels.

-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Micha Feigin]

On Thu, Dec 18, 2003 at 11:41:13PM -0600, Ian Pilcher wrote:
> I posted a couple of weeks ago about a kernel panic that shows up when I
> boot my Abit VP6 (dual-PIII) with ACPI turned on.  It turns out that the
> kernel boots just fine with ACPI on when I build it with
> CONFIG_DEBUG_SLAB=y.
> 
> Alan Cox suggested that this is probably due to code that kmallocs
> memory and then relies on its content without clearing it, but I never
> received any response to my request for suggestions on how to identify
> the offending code.
> 
> I've just spent several hours adding printk statements to the ACPI code,
> and I've reached a point where the oops message is overwriting portions
> of my printk messages.  I have the sinking feeling that this indicates
> that the process executing the ACPI code is not the one that's oops'ing.
> 
> I'm stuck.  How can a relative newbie debug a problem like this, or at
> least gather enough information to enable someone with more expertise to
> do so?
> 

I had a brief look at what CONFIG_DEBUG_SLAB=y option does. It looks
that beside adding some debug output and sanity checks it also
initializes the memory to a non null pattern (poison something). You
could start with setting that pattern to 0x00 instead of what it is and
see if that returns your oopses (don't know where it points, but thats
probably won't do anything but could give you a null pointer access).
Otherwise I would follow the changes (it changes the DEBUG flag and
don't remember what else) and see if it changes anything (other then
the poison thing) or just read tests the results and prints error
messages. If that is the case than what you probably have is a race
condition causing some kind of timing problem.

> TIA.
> -- 
> ========================================================================
> Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
> ========================================================================
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IBM Linux Tutorials.
> Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
> Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
> Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
> _______________________________________________
> Acpi-devel mailing list
> Acpi-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/acpi-devel
> 


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Randy.Dunlap]

On Sat, 20 Dec 2003 13:47:12 -0600 Ian Pilcher <i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org> wrote:

| Randy.Dunlap wrote:
| > 
| > and exactly what kernel version is this?
| > 
| 
| 2.4.23 with the 20031203 ACPI patch.  I get basically the same behavior
| with unpatched 2.4.23 or any of the Fedora Core kernels.

Good.  I only have unpatched 2.4.23 available currently (or 2.6.x).

Is there some reason that you build a kernel with both APM and ACPI
support?  Can you test it without APM support, to see if it changes any?

The dmesg files aren't complete boot logs.  Can you provide complete
boot logs for these 2 cases (panic & hang)?

--
~Randy


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Re: Plea for help

[Ian Pilcher]

Randy.Dunlap wrote:
> Is there some reason that you build a kernel with both APM and ACPI
> support?  Can you test it without APM support, to see if it changes any?

Sorry about the delay.  I had to get all of these kernels, config files,
etc. under control.  (I now have a new appreciation for EXTRAVERSION.)
As a result, I've got three completely new sets of data for your holiday
enjoyment.

The starting point is the i686-smp kernel configuration from Fedora Core
1 (which has both APM and ACPI turned on).  I've built kernel 2.4.23
with ACPI 20031203 with this configuration and called it
2.4.23fc_config.  Then I disabled APM support and built 2.4.23no_apm.
Finally, I turned on frame pointers and build 2.4.23frame_ptrs.

The kernel configs, System.map files, and boot logs for all three of
these kernel are in http://home.comcast.net/~i.pilcher/panics.tar.gz.

> The dmesg files aren't complete boot logs.  Can you provide complete
> boot logs for these 2 cases (panic & hang)?

I knew I wasn't going to get through this without buying a null-modem
cable.  The boot logs in the above tarball were all captured on my
laptop; I hope that's what you're looking for.  Unfortunately, the text
of the panic message is garbled in several of the logs.

I also found that the behavior changes depending on whether I use a
local console (in addition to the serial console) or not.  In the case
of the fc_config kernel, the system actually boots successfully if I use
*only* a serial console.  Check the kernel command line in each boot log
to see what's what.

Am I crazy to think that this almost has to be a race condition?

I'll be out of town until the 29th without access to this system.  I
want to say thanks for taking the time to respond to my "plea".

Happy Holidays!

-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

RE: Plea for help

[Yu, Luming]

>> Is there some reason that you build a kernel with both APM and ACPI
>> support?  Can you test it without APM support, to see if it changes
any?

I want to you build a kernel with CONFIG_ACPI_DEBUG , which could show 
some ACPI debug message that could help us to roughly determine what
could
cause panic ! 



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id\x1278&alloc_id371&op=click

Re: Plea for help

[Ian Pilcher]

Yu, Luming wrote:
> I want to you build a kernel with CONFIG_ACPI_DEBUG , which could show 
> some ACPI debug message that could help us to roughly determine what
> could
> cause panic ! 

I have added a new set of files (acpi_debug) to

     http://home.comcast.net/~i.pilcher/panics.tar.gz

-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

RE: Plea for help

[Yu, Luming]

>Yu, Luming wrote:
> I want to you build a kernel with CONFIG_ACPI_DEBUG , which could show

> some ACPI debug message that could help us to roughly determine what
> could
> cause panic ! 
>
>I have added a new set of files (acpi_debug) to
>
>     http://home.comcast.net/~i.pilcher/panics.tar.gz

I'm not sure, but the most possibility is at acpi_ev_create_gpe_block.
Normally, the next expected message  would be like:

evxfevnt-0093 [04] acpi_enable           : Transition to ACPI mode
successful
evgpeblk-0747 [06] ev_create_gpe_block   : GPE 00 to 15 [_GPE] 2 regs at
0000009

I did find the desultory words showing that in ACPI-DEBUG dmesg.  Is it
possible to dig into ?
BTW, I want to know whether that panic could be found in UP kernel.

Thanks,
Luming 





-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id\x1278&alloc_id371&op=click

Re: Plea for help

[Ian Pilcher]

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]

Yu, Luming wrote:
> I'm not sure, but the most possibility is at acpi_ev_create_gpe_block.
> Normally, the next expected message  would be like:
> 
> evxfevnt-0093 [04] acpi_enable           : Transition to ACPI mode
> successful
> evgpeblk-0747 [06] ev_create_gpe_block   : GPE 00 to 15 [_GPE] 2 regs at
> 0000009
> 
> I did find the desultory words showing that in ACPI-DEBUG dmesg.  Is it
> possible to dig into ?

The attached file is what appears on the local console (modulo possible
typos).  You can see that these messages do appear, mixed in with the
panic information.

> BTW, I want to know whether that panic could be found in UP kernel.

The panic does not occur with a uniprocessor kernel, nor does it happen
when I boot with 'maxcpus=1'.

And just for grins and giggles, I tried booting with 'idle=poll'.  Sure
enough, no panic.

Happy New Year!

-- 
========================================================================
Ian Pilcher                                        i.pilcher-Wuw85uim5zDR7s880joybQ@public.gmane.org
========================================================================

[-- Attachment #2: acpi_debug-console.txt --]
[-- Type: text/plain, Size: 1400 bytes --]

PCI: PCI BIOS revision 2.10 entry at 0xfb370, last bus=1
PCI: Using configuration type 1
 tbxface-0117 [03] acpi_load_tables      : ACPI Tables successfully acquired
Parsing all Control Methods:.....................................................................
..................
Table [DSDT](id F004) - 295 Objects with 29 Devices 70 Methods 19 Regions
ACPI Namespace successfully loaded at root c039949c
IOAPIC[0]: Set PCI routing entry (2-9 -> 0x71 -> IRQ 9 Mode:1 Active:1)
Invalid operand: 0000
<4>evxfevnt-0093 [04] acpi_enable           : CPU:    1
EIP:    0010:[<dffe9fc2>]    Not tainted
<4>Transition to ACPI mode successful
EFLAGS: 00010202
evgpeblk-0747 eax: c0107290   ebx: 5d90f4fb   ecx: 00000000   edx: dffe8000
<4>[06] ev_create_gpe_block   : GPE 00 to 15 [_GPE] 2 regs at 0000000000004020 on int 9
esi: dffe8000   edi: dffe8000   ebp: dffe9fac   esp: dffe9fb0
ds: 0018   es: 0018   ss: 0018
Process swapper (pid: 0, stackpage=dffe9000)
Stack: c0107352 00000002 00000000 00000000 dffe93f8 ffffffff 00000286 00000292
       fffff11d 00000001 00000286 00000000 c0389a2a 00000246 0000002a c0339f8c
       c011f8c3 0000000a 00000000 c02aefa7
Call Trace:    [<c0107352>] [<c011f8c3>]

Code: fe df ff ff ff ff 86 02 00 00 92 02 00 00 1d f1 ff ff 01 00
 <0>Kernel panic: Attempted to kill the idel task!
<4>Completing Region/Field/Buffer/Package initialization:In idle task - not syncing

RE: Plea for help

[Yu, Luming]

>>Yu, Luming wrote:
>> I'm not sure, but the most possibility is at
acpi_ev_create_gpe_block.
>> Normally, the next expected message  would be like:
>> 
>> evxfevnt-0093 [04] acpi_enable           : Transition to ACPI mode
>> successful
>> evgpeblk-0747 [06] ev_create_gpe_block   : GPE 00 to 15 [_GPE] 2 regs
at
>> 0000009
>> 
>> I did find the desultory words showing that in ACPI-DEBUG dmesg.  Is
it
>> possible to dig into ?
>
>The attached file is what appears on the local console (modulo possible
>typos).  You can see that these messages do appear, mixed in with the
>panic information.

Since I cannot reporduce it on my side. Would you please dig into 
function acpi_ev_create_gpe_block.


>> BTW, I want to know whether that panic could be found in UP kernel.
>
>The panic does not occur with a uniprocessor kernel, nor does it happen
>when I boot with 'maxcpus=1'.

Hmm, It sounds like SMP relative issue.

>And just for grins and giggles, I tried booting with 'idle=poll'.  Sure
>enough, no panic.

I'm confused! 

>Happy New Year!

Happy New Year!

--Luming


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id\x1278&alloc_id371&op=click