2.6.19-rc3: more DWARFs and strange messages

2.6.19-rc3: more DWARFs and strange messages

[Martin Lorenz]

With my recent kernel pulled from git on friday I see quite some DWARFs and
other strange messages.

lots of those:
[18038.721000] thinkpad_ec: thinkpad_ec_request_row: bad end STR3:
(0x11:0x00)->0x80

this was said to be triggered by the combination of hdaps and tp_smapi but I
do not load the tp_smapi module ...
hdaps is configured

and quite a few of those:

[18504.980000] BUG: warning at kernel/cpu.c:56/unlock_cpu_hotplug()
[18504.980000]  [<c0103bdd>] dump_trace+0x69/0x1af
[18504.980000]  [<c0103d3b>] show_trace_log_lvl+0x18/0x2c
[18504.980000]  [<c01043da>] show_trace+0xf/0x11
[18504.980000]  [<c01044dd>] dump_stack+0x15/0x17
[18504.980000]  [<c0135e94>] unlock_cpu_hotplug+0x3d/0x66
[18504.980000]  [<f92e67f3>] do_dbs_timer+0x1c2/0x229 [cpufreq_ondemand]
[18504.980000]  [<c012ccb1>] run_workqueue+0x83/0xc5
[18504.980000]  [<c012d5d5>] worker_thread+0xd9/0x10c
[18504.980000]  [<c012fb36>] kthread+0xc2/0xf0
[18504.980000]  [<c010398b>] kernel_thread_helper+0x7/0x10
[18504.980000] DWARF2 unwinder stuck at kernel_thread_helper+0x7/0x10
[18504.980000]
[18504.980000] Leftover inexact backtrace:
[18504.980000]
[18504.980000]  =======================


full config and dmesg is in
http://www.lorenz.eu.org/~mlo/kernel/config.2.6.19-rc3-e1-ie-tp-43.3+1757-g18462d6b-dirty.gz
http://www.lorenz.eu.org/~mlo/kernel/dmesg.2.6.19-rc3-e1-ie-tp-43.3+1757-g18462d6b-dirty.boot
http://www.lorenz.eu.org/~mlo/kernel/dmesg.2.6.19-rc3-e1-ie-tp-43.3+1757-g18462d6b-dirty.run

and more
http://www.lorenz.eu.org/~mlo/kernel/?C=M;O=D

greets
  mlo
--
Dipl.-Ing. Martin Lorenz

            They that can give up essential liberty 
	    to obtain a little temporary safety 
	    deserve neither liberty nor safety.
                                   Benjamin Franklin

please encrypt your mail to me
GnuPG key-ID: F1AAD37D
get it here:
http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xF1AAD37D

ICQ UIN: 33588107

Re: 2.6.19-rc3: more DWARFs and strange messages

[Andi Kleen]

Martin Lorenz <martin@lorenz.eu.org> writes:

> and quite a few of those:
> 
> [18504.980000] BUG: warning at kernel/cpu.c:56/unlock_cpu_hotplug()
> [18504.980000]  [<c0103bdd>] dump_trace+0x69/0x1af
> [18504.980000]  [<c0103d3b>] show_trace_log_lvl+0x18/0x2c
> [18504.980000]  [<c01043da>] show_trace+0xf/0x11
> [18504.980000]  [<c01044dd>] dump_stack+0x15/0x17
> [18504.980000]  [<c0135e94>] unlock_cpu_hotplug+0x3d/0x66
> [18504.980000]  [<f92e67f3>] do_dbs_timer+0x1c2/0x229 [cpufreq_ondemand]
> [18504.980000]  [<c012ccb1>] run_workqueue+0x83/0xc5
> [18504.980000]  [<c012d5d5>] worker_thread+0xd9/0x10c
> [18504.980000]  [<c012fb36>] kthread+0xc2/0xf0
> [18504.980000]  [<c010398b>] kernel_thread_helper+0x7/0x10
> [18504.980000] DWARF2 unwinder stuck at kernel_thread_helper+0x7/0x10

What gcc / binutils version do you use?

> [18504.980000]
> [18504.980000] Leftover inexact backtrace:
> [18504.980000]
> [18504.980000]  =======================
> 

-Andi

Re: 2.6.19-rc3: more DWARFs and strange messages

[Martin Lorenz]

On Tue, Oct 31, 2006 at 03:31:41PM +0100, Andi Kleen wrote:
> Martin Lorenz <martin@lorenz.eu.org> writes:
> 
> > and quite a few of those:
> > 
> > [18504.980000] BUG: warning at kernel/cpu.c:56/unlock_cpu_hotplug()
> > [18504.980000]  [<c0103bdd>] dump_trace+0x69/0x1af
> > [18504.980000]  [<c0103d3b>] show_trace_log_lvl+0x18/0x2c
> > [18504.980000]  [<c01043da>] show_trace+0xf/0x11
> > [18504.980000]  [<c01044dd>] dump_stack+0x15/0x17
> > [18504.980000]  [<c0135e94>] unlock_cpu_hotplug+0x3d/0x66
> > [18504.980000]  [<f92e67f3>] do_dbs_timer+0x1c2/0x229 [cpufreq_ondemand]
> > [18504.980000]  [<c012ccb1>] run_workqueue+0x83/0xc5
> > [18504.980000]  [<c012d5d5>] worker_thread+0xd9/0x10c
> > [18504.980000]  [<c012fb36>] kthread+0xc2/0xf0
> > [18504.980000]  [<c010398b>] kernel_thread_helper+0x7/0x10
> > [18504.980000] DWARF2 unwinder stuck at kernel_thread_helper+0x7/0x10
> 
> What gcc / binutils version do you use?
$ gcc --version
gcc (GCC) 4.1.2 20061020 (prerelease) (Debian 4.1.1-17)

dpkg says:
ii  gcc                        4.1.1-13

$ ar --version
GNU ar 2.17 Debian GNU/Linux

dpkg says:
ii  binutils                   2.17-3


> 
> > [18504.980000]
> > [18504.980000] Leftover inexact backtrace:
> > [18504.980000]
> > [18504.980000]  =======================
> > 
> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

gruss
  mlo
--
Dipl.-Ing. Martin Lorenz

            They that can give up essential liberty 
	    to obtain a little temporary safety 
	    deserve neither liberty nor safety.
                                   Benjamin Franklin

please encrypt your mail to me
GnuPG key-ID: F1AAD37D
get it here:
http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xF1AAD37D

ICQ UIN: 33588107

Re: 2.6.19-rc3: more DWARFs and strange messages

[Jan Beulich]

Can you perhaps get us arch/i386/kernel/{entry,process}.o,
.config, and (assuming you can reproduce the original problem)
the raw stack dump obtained with a sufficiently high kstack=
option?

Thanks, Jan

>>> Martin Lorenz <martin@lorenz.eu.org> 31.10.06 17:08 >>>
On Tue, Oct 31, 2006 at 03:31:41PM +0100, Andi Kleen wrote:
> Martin Lorenz <martin@lorenz.eu.org> writes:
> 
> > and quite a few of those:
> > 
> > [18504.980000] BUG: warning at kernel/cpu.c:56/unlock_cpu_hotplug()
> > [18504.980000]  [<c0103bdd>] dump_trace+0x69/0x1af
> > [18504.980000]  [<c0103d3b>] show_trace_log_lvl+0x18/0x2c
> > [18504.980000]  [<c01043da>] show_trace+0xf/0x11
> > [18504.980000]  [<c01044dd>] dump_stack+0x15/0x17
> > [18504.980000]  [<c0135e94>] unlock_cpu_hotplug+0x3d/0x66
> > [18504.980000]  [<f92e67f3>] do_dbs_timer+0x1c2/0x229 [cpufreq_ondemand]
> > [18504.980000]  [<c012ccb1>] run_workqueue+0x83/0xc5
> > [18504.980000]  [<c012d5d5>] worker_thread+0xd9/0x10c
> > [18504.980000]  [<c012fb36>] kthread+0xc2/0xf0
> > [18504.980000]  [<c010398b>] kernel_thread_helper+0x7/0x10
> > [18504.980000] DWARF2 unwinder stuck at kernel_thread_helper+0x7/0x10
> 
> What gcc / binutils version do you use?
$ gcc --version
gcc (GCC) 4.1.2 20061020 (prerelease) (Debian 4.1.1-17)

dpkg says:
ii  gcc                        4.1.1-13

$ ar --version
GNU ar 2.17 Debian GNU/Linux

dpkg says:
ii  binutils                   2.17-3


> 
> > [18504.980000]
> > [18504.980000] Leftover inexact backtrace:
> > [18504.980000]
> > [18504.980000]  =======================
> > 
> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at  http://vger.kernel.org/majordomo-info.html 
> Please read the FAQ at  http://www.tux.org/lkml/ 

gruss
  mlo
--
Dipl.-Ing. Martin Lorenz

            They that can give up essential liberty 
	    to obtain a little temporary safety 
	    deserve neither liberty nor safety.
                                   Benjamin Franklin

please encrypt your mail to me
GnuPG key-ID: F1AAD37D
get it here:
http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xF1AAD37D 

ICQ UIN: 33588107

Re: 2.6.19-rc3: more DWARFs and strange messages

[Andi Kleen]

On Tuesday 31 October 2006 17:28, Jan Beulich wrote:
> Can you perhaps get us arch/i386/kernel/{entry,process}.o,
> .config, and (assuming you can reproduce the original problem)
> the raw stack dump obtained with a sufficiently high kstack=
> option?

WARN_ON unfortunately doesn't dump the raw stack at all (maybe that
should be fixed) 

-Andi

Re: 2.6.19-rc3: more DWARFs and strange messages

[Martin Lorenz]

On Tue, Oct 31, 2006 at 06:28:52PM +0100, Andi Kleen wrote:
> On Tuesday 31 October 2006 17:28, Jan Beulich wrote:
> > Can you perhaps get us arch/i386/kernel/{entry,process}.o,
> > .config, and (assuming you can reproduce the original problem)
> > the raw stack dump obtained with a sufficiently high kstack=
> > option?
> 
> WARN_ON unfortunately doesn't dump the raw stack at all (maybe that
> should be fixed) 

what is a reasonable kstack parameter to be informative for you?

> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

gruss
  mlo
--
Dipl.-Ing. Martin Lorenz

            They that can give up essential liberty 
	    to obtain a little temporary safety 
	    deserve neither liberty nor safety.
                                   Benjamin Franklin

please encrypt your mail to me
GnuPG key-ID: F1AAD37D
get it here:
http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xF1AAD37D

ICQ UIN: 33588107

Re: 2.6.19-rc3: more DWARFs and strange messages

[Jan Beulich]

>what is a reasonable kstack parameter to be informative for you?

This unfortunately depends on the depth of the stack that is in use
at the point the dump is taken. The only safe value would be to
dump the full stack size (kstack=1024 for 4k stack, kstack=2048
for 8k ones), but since it'll stop at a stack boundary perhaps that's
what you should go with.

As to Andi's remark regarding WARN_ON() - you'd have to address
that issue in a private patch first, or the addition of the kstack=
parameter will be useless. I presume it's likely you don't have the
time to do that...

Jan

Re: 2.6.19-rc3: more DWARFs and strange messages

[Martin Lorenz]

On Thu, Nov 02, 2006 at 09:37:58AM +0000, Jan Beulich wrote:
> >what is a reasonable kstack parameter to be informative for you?
> 
> This unfortunately depends on the depth of the stack that is in use
> at the point the dump is taken. The only safe value would be to
> dump the full stack size (kstack=1024 for 4k stack, kstack=2048
> for 8k ones), but since it'll stop at a stack boundary perhaps that's
> what you should go with.
> 
> As to Andi's remark regarding WARN_ON() - you'd have to address
> that issue in a private patch first, or the addition of the kstack=
> parameter will be useless. I presume it's likely you don't have the
> time to do that...

well...
to be honest, I need my computer to work during usual daily business
at the moment I can afford to do a compile and test a new kernel once a day,
but if it introduces instabilities that make my computer unusable for
productive work (which means programming, running heavy java applications
and  and TeXing) I can't run it for more than a few minutes.

I started a bisection last weekend but had to give up after three
compile-reboot cycles.

so, please don't expect too much, but I will do my best to help

gruss
  mlo
--
Dipl.-Ing. Martin Lorenz

            They that can give up essential liberty 
	    to obtain a little temporary safety 
	    deserve neither liberty nor safety.
                                   Benjamin Franklin

please encrypt your mail to me
GnuPG key-ID: F1AAD37D
get it here:
http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xF1AAD37D

ICQ UIN: 33588107

Re: 2.6.19-rc3: more DWARFs and strange messages

[Jan Beulich]

Hmm, another thing you could do is a sysrq-t with the increased
kstack= - this should encounter several kernel_thread()-s, and all
of them should exhibit the problem you were seeing. That output
could then be of help.

Thanks, Jan

Re: 2.6.19-rc3: more DWARFs and strange messages

[Adrian Bunk]

Martin, did commit 4b96b1a10cb00c867103b21f0f2a6c91b705db11 that is now 
in Linus' tree fix this issue?


On Sat, Oct 28, 2006 at 10:01:51PM +0200, Martin Lorenz wrote:
> With my recent kernel pulled from git on friday I see quite some DWARFs and
> other strange messages.
> 
> lots of those:
> [18038.721000] thinkpad_ec: thinkpad_ec_request_row: bad end STR3:
> (0x11:0x00)->0x80
> 
> this was said to be triggered by the combination of hdaps and tp_smapi but I
> do not load the tp_smapi module ...
> hdaps is configured
> 
> and quite a few of those:
> 
> [18504.980000] BUG: warning at kernel/cpu.c:56/unlock_cpu_hotplug()
> [18504.980000]  [<c0103bdd>] dump_trace+0x69/0x1af
> [18504.980000]  [<c0103d3b>] show_trace_log_lvl+0x18/0x2c
> [18504.980000]  [<c01043da>] show_trace+0xf/0x11
> [18504.980000]  [<c01044dd>] dump_stack+0x15/0x17
> [18504.980000]  [<c0135e94>] unlock_cpu_hotplug+0x3d/0x66
> [18504.980000]  [<f92e67f3>] do_dbs_timer+0x1c2/0x229 [cpufreq_ondemand]
> [18504.980000]  [<c012ccb1>] run_workqueue+0x83/0xc5
> [18504.980000]  [<c012d5d5>] worker_thread+0xd9/0x10c
> [18504.980000]  [<c012fb36>] kthread+0xc2/0xf0
> [18504.980000]  [<c010398b>] kernel_thread_helper+0x7/0x10
> [18504.980000] DWARF2 unwinder stuck at kernel_thread_helper+0x7/0x10
> [18504.980000]
> [18504.980000] Leftover inexact backtrace:
> [18504.980000]
> [18504.980000]  =======================
> 
> 
> full config and dmesg is in
> http://www.lorenz.eu.org/~mlo/kernel/config.2.6.19-rc3-e1-ie-tp-43.3+1757-g18462d6b-dirty.gz
> http://www.lorenz.eu.org/~mlo/kernel/dmesg.2.6.19-rc3-e1-ie-tp-43.3+1757-g18462d6b-dirty.boot
> http://www.lorenz.eu.org/~mlo/kernel/dmesg.2.6.19-rc3-e1-ie-tp-43.3+1757-g18462d6b-dirty.run
> 
> and more
> http://www.lorenz.eu.org/~mlo/kernel/?C=M;O=D
> 
> greets
>   mlo
> --
> Dipl.-Ing. Martin Lorenz
> 
>             They that can give up essential liberty 
> 	    to obtain a little temporary safety 
> 	    deserve neither liberty nor safety.
>                                    Benjamin Franklin
> 
> please encrypt your mail to me
> GnuPG key-ID: F1AAD37D
> get it here:
> http://blackhole.pca.dfn.de:11371/pks/lookup?op=get&search=0xF1AAD37D
> 
> ICQ UIN: 33588107

Re: 2.6.19-rc3: more DWARFs and strange messages

[Jan Beulich]

>>> Martin Lorenz <martin@lorenz.eu.org> 31.10.06 18:03 >>>
>On Tue, Oct 31, 2006 at 05:28:11PM +0100, Jan Beulich wrote:
>> Can you perhaps get us arch/i386/kernel/{entry,process}.o,
>> .config, and (assuming you can reproduce the original problem)
>> the raw stack dump obtained with a sufficiently high kstack=
>> option?
>
>config and the requested .o files are attached
>...hoping I diden't loose track of my kernel bilds
>stacktrace will follow ASAP

I'm unable to find anything pointing out the reason for this
misbehavior, and I continue to be unable to reproduce it on my
test systems.
Short of somebody else wanting to try to debug this, perhaps
the only way to get more insight is to add some more debug
printing to the unwinder and/or its callers, which I don't have
time to do immediately.

Jan