Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Gabriel C]

Danny ter Haar wrote:
[ added  linux-acpi and Len to CC ]
> Quoting Gabriel C (nix.or.die@googlemail.com):
>> Maybe try to : 
>> disable BSG ( maybe some leftover bug )
>> boot acpi=off ( that got merged kind late )
> 
> My first git disected kernel wouldn't boot, but with
> acpi=off it would indeed boot!

Now while we think is ACPI this should be easy for you to bisect.

This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
merged ACPI so this one should be your first bad one.

Maybe Len has some idea and you don't need to bisect :)

> 
> As did the 2.6.23-rc1-git5 kernel...
> 
> I will bisect further to find out exactly what patch is
> playing up in my particular setup.
> 
> thanks for the tip! ;-)

You are welcome :)

> 
> Danny
> 

Gabriel

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Gabriel C (nix.or.die@googlemail.com):
> Now while we think is ACPI this should be easy for you to bisect.
> This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
> merged ACPI so this one should be your first bad one.
> Maybe Len has some idea and you don't need to bisect :)

Thanks to personal coaching of Gabriel i bisected the last few days.

It looked like this was the cullprit:

22aadf8a07067644e101267ed5003043f2ad05bf is first bad commit

2.6.23-rc1-git[1-6] all lockup solid after either direct or
within a couple of minutes (less than 2) after reboot.

They all run fine with "acpi=off" as boot argument.


However, i'm currently running 2.6.23-rc1-git6 with this reverted:

diff --git a/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c b/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c
index 18c8b67..6f846be 100644 (file)
--- a/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c
+++ b/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c
@@ -665,8 +665,8 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
        data->max_freq = perf->states[0].core_frequency * 1000;
        /* table init */
        for (i=0; i<perf->state_count; i++) {
-               if (i>0 && perf->states[i].core_frequency ==
-                   perf->states[i-1].core_frequency)
+               if (i>0 && perf->states[i].core_frequency >=
+                   data->freq_table[valid_states-1].frequency / 1000)
                        continue;

-----------

I could hardly believe this could be the cause.
And indeed after about 16 minutes the kernel froze again, though longer
uptime than other kernels :-(

This hardware is a via epia 5000 with latest bios available (2.07)

It's remarkable that with acpi=off the machine is rocksolid.
As is with 2.6.22* kernels _with_ acpi enabled!
weeks of uptime before i wanted to upgrade to a new kernel ;-)

I'm thinking of redoing the git disect but this time really powercycle
the unit between kernels since it "seems/feels" like a timer which 
really counts to 0 and then locks the machine. 
After a lockup/freeze i cant boot the kernel another time with acpi 
enabled: it will simply hang after booting init.
And again nobody home: no keyboard activity whatsoever.

I've put the dmesg of 2.6.23-rc1-git6 with and without acpi=off online:
http://www.dth.net/kernel/via_output_2.6.23-rc1-git6-acpi_off
http://www.dth.net/kernel/via_output_2.6.23-rc1-git6-acpi_on

I would like to hear if someone has an idea how to tackle this problem.

Danny
-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Len Brown]

On Sunday 29 July 2007 22:54, Danny ter Haar wrote:
> Quoting Gabriel C (nix.or.die@googlemail.com):
> > Now while we think is ACPI this should be easy for you to bisect.
> > This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
> > merged ACPI so this one should be your first bad one.
> > Maybe Len has some idea and you don't need to bisect :)
> 
> Thanks to personal coaching of Gabriel i bisected the last few days.
> 
> It looked like this was the cullprit:
> 
> 22aadf8a07067644e101267ed5003043f2ad05bf is first bad commit

Please attach the output from acpidump to 
http://bugzilla.kernel.org/show_bug.cgi?id=7880
We'll likely be able to tell from it if that patch
has any real effect on your system, or if you test
result was from some unrelated event.

Also, please test with CONFIG_X86_ACPI_CPUFREQ=n
to remove the acpi-cpufreq driver (and thus this patch)
from your kernel.  If it still fails, then we know
that this driver (and this patch) are not related
to the failure.

> 2.6.23-rc1-git[1-6] all lockup solid after either direct or
> within a couple of minutes (less than 2) after reboot.
> 
> They all run fine with "acpi=off" as boot argument.

Hmmm, okay, the "big hammer" works.  Please see
if any of these smaller hammers work:

pnpacpi=off
acpi=noirq
notsc

thanks,
-Len

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Please attach the output from acpidump to 
> http://bugzilla.kernel.org/show_bug.cgi?id=7880

done
also on : http://www.dth.net/kernel/acpidump_output_via_c3_5000

> We'll likely be able to tell from it if that patch
> has any real effect on your system, or if you test
> result was from some unrelated event.

Reverting the patch didn't make a real difference, it only took
longer (eg 16 minutes before lockup in stead of 2-3 minutes)

> Also, please test with CONFIG_X86_ACPI_CPUFREQ=n
> to remove the acpi-cpufreq driver (and thus this patch)
> from your kernel.  If it still fails, then we know
> that this driver (and this patch) are not related
> to the failure.

Will do.
This machine(firewall) is solar powered so i tried to let it
consume as less power as possible 
This is in my startup script at the moment:

echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> 
> pnpacpi=off
> acpi=noirq
> notsc

More testing to do, however, sunday i'm going abroad for a while,
all stuff will be in storage. If i cant find in the next few days
(packing everything in moving boxes) it might take a while before
i can test again. Sorry for that.

Danny

-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> 
> pnpacpi=off
Went out for dinner, machine was frozen on return.
One less on the checklist ..

-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> acpi=noirq

died within 2 minutes

> notsc

After 36 minutes

-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Also, please test with CONFIG_X86_ACPI_CPUFREQ=n
> to remove the acpi-cpufreq driver (and thus this patch)
> from your kernel.  If it still fails, then we know
> that this driver (and this patch) are not related
> to the failure.

It wasn't enabled in any of the kernels i ran:

# grep X86_ACPI ../configs/config*
../configs/config-2.6.22-git14-c3-via5000-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.22-git17-c3-via5000-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git5:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git6:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git9:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-mm1-c3-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set

> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> 
> pnpacpi=off
> acpi=noirq
> notsc

_none_ of these options will work. The machine locks up solid.
And that's what i'm the most surprised about. The last time i withnessed
a solid kernel lockup without any panic/notification is _years_ ago.

I also tried a kernel without "CPU Frequency scaling", so it is not the
scaling up/down in Mhz of the cpu.
I've even just compiled/run a kernel with libata and even that one still 
locks up.

New round of git bisects i guess (this time powercycle instead of reboot
and wait few hours before ginving it "bisect good" )

Will report back.

Danny

-- 

Re: Update: ACPI problems: 2.6.22-git17 working, 2.6.23-rc1* is not: (was ide not working)

[Danny ter Haar]

Okay

Redid git bisect from start:

git bisect log
git-bisect start
# good: [098fd16f00005f665d3baa7e682d8cb3d7c0fe6f] Linux 2.6.22
git-bisect good 098fd16f00005f665d3baa7e682d8cb3d7c0fe6f
# good: [098fd16f00005f665d3baa7e682d8cb3d7c0fe6f] Linux 2.6.22
git-bisect good 098fd16f00005f665d3baa7e682d8cb3d7c0fe6f
# bad: [7d57c74238cdf570bca20b711b2c0b31a553c1e5] Linux 2.6.23-rc1
git-bisect bad 7d57c74238cdf570bca20b711b2c0b31a553c1e5
# good: [1f1c2881f673671539b25686df463518d69c4649] Merge branch
# 'upstream-linus' of
# master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/netdev-2.6
git-bisect good 1f1c2881f673671539b25686df463518d69c4649
# good: [5b7f13bd26a0c1d394a1a1f2bb6de5130c3a3843] coda: update module
# information
git-bisect good 5b7f13bd26a0c1d394a1a1f2bb6de5130c3a3843
# good: [74889e41d9a1f80928130a02af9b010673bc5ba7] [CELL] cbe_cpufreq:
# reorganize code
git-bisect good 74889e41d9a1f80928130a02af9b010673bc5ba7
# good: [bdda1561ffda764583a295229db66d94cf6038a3] x86_64: x86_64 - Use
# non locked version for local_cmpxchg()
git-bisect good bdda1561ffda764583a295229db66d94cf6038a3
# bad: [d7f5e3df3574c6e38b99f5fe22f15540b2b9811d] Merge branch
# 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
git-bisect bad d7f5e3df3574c6e38b99f5fe22f15540b2b9811d
# bad: [08e31686d6d119ba26bf0690f5f872f6f5bd1a97] Pull thinkpad into
# release branch
git-bisect bad 08e31686d6d119ba26bf0690f5f872f6f5bd1a97
# bad: [f79e3185dd0f8650022518d7624c876d8929061b] Pull misc into release
# branch
git-bisect bad f79e3185dd0f8650022518d7624c876d8929061b
# good: [22aadf8a07067644e101267ed5003043f2ad05bf] Pull bugzilla-7880
# into release branch
git-bisect good 22aadf8a07067644e101267ed5003043f2ad05bf
# good: [e8b495fe09bc793ae26774e7b2667f7f658d56e2] Pull dock-bay into
# release branch
git-bisect good e8b495fe09bc793ae26774e7b2667f7f658d56e2
# good: [18eab8550397f1f3d4b8b2c5257c88dae25d58ed] ACPI: Enable C3 even
# when PM2_control is zero
git-bisect good 18eab8550397f1f3d4b8b2c5257c88dae25d58ed
# good: [7a9b9068b8c7b526894904c0b9b90354825b8fbd] ACPI: update
# feature-removal-schedule.txt, /sys/firmware/acpi/namespace is gone
git-bisect good 7a9b9068b8c7b526894904c0b9b90354825b8fbd
# good: [8b8eb7d8cfc6cd95ed00cd58754e8493322505bd] ACPI: update ACPI
# proc I/F removal schedule
git-bisect good 8b8eb7d8cfc6cd95ed00cd58754e8493322505bd


f79e3185dd0f8650022518d7624c876d8929061b is first bad commit

I will store this particular firewall setup friday (will go into storage
for 2 months), if somebody would like more info, me to try something,
please send it to me asap.

Thanks for looking at this.

Danny
--