ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[dth]

I have a via mini-itx epia 5000 motherboard as a firewall.
It has an ide device: PQI 128MB flash DOM [1] which just plugs
directly into the ide connector on the motherboard.
2.6.22-git17 works, although it gives some warning 
hda: set_drive_speed_status: status=0x51 { DriveReady SeekComplete Error }
hda: set_drive_speed_status: error=0x04 { DriveStatusError }
Full dmesg output [2]

When i compiled any 2.6.23-rc1 kernel out so far it always froze on me.
Yesterday i tried 2.6.23-rc1-git3 and i booted until:

VP_IDE: VIA vt8231 (rev 10) IDE UDMA100 controller on pci0000:00:11.1
    ide0: BM-DMA at 0xd000-0xd007, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:pio, hdd:pio
Marking TSC unstable due to: possible TSC halt in C2.
Time: acpi_pm clocksource has been installed.
hda: PQI IDE DiskOnModule, ATA DISK drive
hda: IRQ probe failed (0xfff2)
Clocksource tsc unstable (delta = 84358771493 ns)

So i went back to 2.6.22-git17.
This morning i saw a fresh git4 and compiled/installed that.
This kernel actually booted! (once)
I had a netconsole running to catch the lucky event [3]

After about 2 minutes of working however, the whole machine froze.
No message at the console, not able to toggle numlock, no magic
sysrq key features: solid frozen.
After the power cycle i was not able to boot the same kernel
ever again. It was just like a timer had run out and it wouldn't 
work anymore. 

I know my hardware is "ancient" and the flash thing is probably
not current. My firewall however is solar powerpowered (10 watt 
power consumption) and it would be fun if i could run the latest/
greatest kernels on it.

Config files and hdparm info on the DOM in the same directory as
dmesg output.

Any comments appriciated.

Danny

[1] http://www.memorydepot.com/ssd_diskonmodule.asp
[2] http://www.dth.net/kernel/via_output_2.6.22_git17
[3] http://www.dth.net/kernel/via_output_2.6.23-rc1-git4


-- 

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Bartlomiej Zolnierkiewicz]

Hi,

On Friday 27 July 2007, dth wrote:
> I have a via mini-itx epia 5000 motherboard as a firewall.
> It has an ide device: PQI 128MB flash DOM [1] which just plugs
> directly into the ide connector on the motherboard.
> 2.6.22-git17 works, although it gives some warning 
> hda: set_drive_speed_status: status=0x51 { DriveReady SeekComplete Error }
> hda: set_drive_speed_status: error=0x04 { DriveStatusError }

Should be harmless for now but we would like to fix it the long-term,
please send "hdparm --Istdout /dev/hda" output.

> Full dmesg output [2]
> 
> When i compiled any 2.6.23-rc1 kernel out so far it always froze on me.
> Yesterday i tried 2.6.23-rc1-git3 and i booted until:

No IDE changes yet after 2.6.22-git17 so something else must have broke down.

Please retry with the latest -git kernel and if the problem is still
there install git, get kernel tree and run git-bisect.

Since 2.6.22-git17 works fine the initial "good" commit would be
e51f802babc5e368c60fbfd08c6c11269c9253b0 and the initial "bad" one
f695baf2df9e0413d3521661070103711545207a (for 2.6.23-rc1).

If you need some practical examples on using git-bisect, this one

http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find-buggy-kernel-patches/

is a very good one (IMO).

> VP_IDE: VIA vt8231 (rev 10) IDE UDMA100 controller on pci0000:00:11.1
>     ide0: BM-DMA at 0xd000-0xd007, BIOS settings: hda:pio, hdb:pio
>     ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:pio, hdd:pio
> Marking TSC unstable due to: possible TSC halt in C2.
> Time: acpi_pm clocksource has been installed.
> hda: PQI IDE DiskOnModule, ATA DISK drive
> hda: IRQ probe failed (0xfff2)

This message would indicate broken IRQ routing, however it is no
longer present in the log for the kernel 2.6.23-rc1-git4 ([3]).

Interesting that the kernel 2.6.22-git17 (log [2]) doesn't use via82cxxx
IDE host driver while the kernel 2.6.23-rc1-git4 (log [3]) does...?

> Clocksource tsc unstable (delta = 84358771493 ns)
> 
> So i went back to 2.6.22-git17.
> This morning i saw a fresh git4 and compiled/installed that.
> This kernel actually booted! (once)
> I had a netconsole running to catch the lucky event [3]
> 
> After about 2 minutes of working however, the whole machine froze.
> No message at the console, not able to toggle numlock, no magic
> sysrq key features: solid frozen.
> After the power cycle i was not able to boot the same kernel
> ever again. It was just like a timer had run out and it wouldn't 
> work anymore. 
> 
> I know my hardware is "ancient" and the flash thing is probably
> not current. My firewall however is solar powerpowered (10 watt 

Neat. :)

> power consumption) and it would be fun if i could run the latest/
> greatest kernels on it.

With the buggy commit number obtained from git-bisect it should be
fixed quite quickly.

> Config files and hdparm info on the DOM in the same directory as
> dmesg output.
> 
> Any comments appriciated.
> 
> Danny
> 
> [1] http://www.memorydepot.com/ssd_diskonmodule.asp
> [2] http://www.dth.net/kernel/via_output_2.6.22_git17
> [3] http://www.dth.net/kernel/via_output_2.6.23-rc1-git4

Thanks,
Bart

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Danny ter Haar]

Quoting Bartlomiej Zolnierkiewicz (bzolnier@gmail.com):
> Should be harmless for now but we would like to fix it the long-term,
> please send "hdparm --Istdout /dev/hda" output.


I allready made that available in the same subdir:

http://www.dth.net/kernel/

Output repeated here:


voyage:~# hdparm -I /dev/hda

/dev/hda:

ATA device, with non-removable media
        Model Number:       PQI IDE DiskOnModule                    
        Serial Number:      
        Firmware Revision:  db01.20a
Standards:
        Likely used: 1
Configuration:
        hard sectored
        not MFM encoded 
        head switch time > 15us
        fixed drive
        disk xfer rate > 5Mbs
        Logical         max     current
        cylinders       500     500
        heads           16      16
        sectors/track   32      32
        --
        bytes/track: 0  bytes/sector: 528
        CHS current addressable sectors:     256000
        LBA    user addressable sectors:     256000
        device size with M = 1024*1024:         125 MBytes
        device size with M = 1000*1000:         131 MBytes 
Capabilities:
        LBA, IORDY not likely
        Buffer type: 0002: dual port, multi-sector
        Buffer size: 1.0kB      bytes avail on r/w long: 4
        Cannot perform double-word IO
        R/W multiple sector transfer: Max = 1   Current = 0
        DMA: not supported
        PIO: pio0 pio1 pio2 

voyage:~# hdparm -tT /dev/hda
/dev/hda:
 Timing cached reads:   116 MB in  2.00 seconds =  57.92 MB/sec
 Timing buffered disk reads:   16 MB in  3.01 seconds =   5.32 MB/sec
# hdparm --Istdout /dev/hda

/dev/hda:
045a 01f4 0000 0010 0000 0210 0020 0003
e800 0000 2020 2020 2020 2020 2020 2020
2020 2020 2020 2020 0002 0002 0004 6462
3031 2e32 3061 5051 4920 4944 4520 4469
736b 4f6e 4d6f 6475 6c65 2020 2020 2020
2020 2020 2020 2020 2020 2020 2020 0001
0000 0200 0000 0200 0000 0001 01f4 0010
0020 e800 0003 0100 e800 0003 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000

> No IDE changes yet after 2.6.22-git17 so something else must have broke down.

darn....

I also tried with libata driver on the 2.6.22 series but that wouldn't 
work as well. I contacted Alan Cox and he made some patches/suggestions
that made 2.6.22-git16 with libata work!
I hope his patch will make it into mainline soon.

For the dmesg output of this kernel 
http://www.dth.net/kernel/via_output_2.6.22-git16-libata_fix_alan_cox

> Please retry with the latest -git kernel and if the problem is still
> there install git, get kernel tree and run git-bisect.
> 
> Since 2.6.22-git17 works fine the initial "good" commit would be
> e51f802babc5e368c60fbfd08c6c11269c9253b0 and the initial "bad" one
> f695baf2df9e0413d3521661070103711545207a (for 2.6.23-rc1).
> 
> If you need some practical examples on using git-bisect, this one
> 
> http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find-buggy-kernel-patches/
> 
> is a very good one (IMO).

I will certainly will read/try that.

> > VP_IDE: VIA vt8231 (rev 10) IDE UDMA100 controller on pci0000:00:11.1
> >     ide0: BM-DMA at 0xd000-0xd007, BIOS settings: hda:pio, hdb:pio
> >     ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:pio, hdd:pio
> > Marking TSC unstable due to: possible TSC halt in C2.
> > Time: acpi_pm clocksource has been installed.
> > hda: PQI IDE DiskOnModule, ATA DISK drive
> > hda: IRQ probe failed (0xfff2)
> 
> This message would indicate broken IRQ routing, however it is no
> longer present in the log for the kernel 2.6.23-rc1-git4 ([3]).

i booted with routeirq=pci but gave the same negative result.
 
> Interesting that the kernel 2.6.22-git17 (log [2]) doesn't use via82cxxx
> IDE host driver while the kernel 2.6.23-rc1-git4 (log [3]) does...?

kernel configs are also in above mentioned webdir

basically copied the 2.6.22-git17 and ran a make oldconfig.


Did a diff -u on those config files and there is really a lot different:

[snip]
-# ATA/IDE/MFM/RLL support
+# Device Drivers
 #
[snip]
 #
 # IDE chipset support/bugfixes
 #
+# CONFIG_BLK_DEV_TRM290 is not set
 CONFIG_BLK_DEV_VIA82CXXX=y
-# CONFIG_IDE_CHIPSETS is not set
-# CONFIG_IDEDMA_AUTO is not set
+# CONFIG_BLK_DEV_TC86C001 is not set
+# CONFIG_IDE_ARM is not set
+CONFIG_BLK_DEV_IDEDMA=y
 # CONFIG_IDEDMA_IVB is not set
-# CONFIG_DMA_NONPCI is not set


I will run a make menuconfig and see if i can clean things up.

> > I know my hardware is "ancient" and the flash thing is probably
> > not current. My firewall however is solar powerpowered (10 watt 
> 
> Neat. :)
> 
> > power consumption) and it would be fun if i could run the latest/
> > greatest kernels on it.
> 
> With the buggy commit number obtained from git-bisect it should be
> fixed quite quickly.


Thanks for your feedback/enthousiasm!

Danny
-- 

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Danny ter Haar]

Quoting Bartlomiej Zolnierkiewicz (bzolnier@gmail.com):
> Please retry with the latest -git kernel and if the problem is still
> there install git, get kernel tree and run git-bisect.

I ran over "make menuconfig" and did a few changes.

http://www.dth.net/kernel/config-2.6.23-rc1-git5

It boots, but freezes solid before doing any "work" as a firewall.
I was able to catch boot messages with netconsole

http://www.dth.net/kernel/via_output_2.6.23-rc1-git5

It didn't respond after the last line.
magig sysrq etc, all nada.

Will start to get acquainted with git ;-)

Danny
-- 

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Gabriel C]

Danny ter Haar wrote:
> Quoting Bartlomiej Zolnierkiewicz (bzolnier@gmail.com):
>> Please retry with the latest -git kernel and if the problem is still
>> there install git, get kernel tree and run git-bisect.
> 
> I ran over "make menuconfig" and did a few changes.
> 
> http://www.dth.net/kernel/config-2.6.23-rc1-git5
> 

Maybe try to : 

disable BSG ( maybe some leftover bug )
boot acpi=off ( that got merged kind late )

> 
> Danny


Gabriel

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Danny ter Haar]

Quoting Gabriel C (nix.or.die@googlemail.com):
> Maybe try to : 
> disable BSG ( maybe some leftover bug )
> boot acpi=off ( that got merged kind late )

My first git disected kernel wouldn't boot, but with
acpi=off it would indeed boot!

As did the 2.6.23-rc1-git5 kernel...

I will bisect further to find out exactly what patch is
playing up in my particular setup.

thanks for the tip! ;-)

Danny

-- 

Re: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not

[Gabriel C]

Danny ter Haar wrote:
[ added  linux-acpi and Len to CC ]
> Quoting Gabriel C (nix.or.die@googlemail.com):
>> Maybe try to : 
>> disable BSG ( maybe some leftover bug )
>> boot acpi=off ( that got merged kind late )
> 
> My first git disected kernel wouldn't boot, but with
> acpi=off it would indeed boot!

Now while we think is ACPI this should be easy for you to bisect.

This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
merged ACPI so this one should be your first bad one.

Maybe Len has some idea and you don't need to bisect :)

> 
> As did the 2.6.23-rc1-git5 kernel...
> 
> I will bisect further to find out exactly what patch is
> playing up in my particular setup.
> 
> thanks for the tip! ;-)

You are welcome :)

> 
> Danny
> 

Gabriel

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Gabriel C (nix.or.die@googlemail.com):
> Now while we think is ACPI this should be easy for you to bisect.
> This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
> merged ACPI so this one should be your first bad one.
> Maybe Len has some idea and you don't need to bisect :)

Thanks to personal coaching of Gabriel i bisected the last few days.

It looked like this was the cullprit:

22aadf8a07067644e101267ed5003043f2ad05bf is first bad commit

2.6.23-rc1-git[1-6] all lockup solid after either direct or
within a couple of minutes (less than 2) after reboot.

They all run fine with "acpi=off" as boot argument.


However, i'm currently running 2.6.23-rc1-git6 with this reverted:

diff --git a/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c b/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c
index 18c8b67..6f846be 100644 (file)
--- a/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c
+++ b/arch/i386/kernel/cpu/cpufreq/acpi-cpufreq.c
@@ -665,8 +665,8 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
        data->max_freq = perf->states[0].core_frequency * 1000;
        /* table init */
        for (i=0; i<perf->state_count; i++) {
-               if (i>0 && perf->states[i].core_frequency ==
-                   perf->states[i-1].core_frequency)
+               if (i>0 && perf->states[i].core_frequency >=
+                   data->freq_table[valid_states-1].frequency / 1000)
                        continue;

-----------

I could hardly believe this could be the cause.
And indeed after about 16 minutes the kernel froze again, though longer
uptime than other kernels :-(

This hardware is a via epia 5000 with latest bios available (2.07)

It's remarkable that with acpi=off the machine is rocksolid.
As is with 2.6.22* kernels _with_ acpi enabled!
weeks of uptime before i wanted to upgrade to a new kernel ;-)

I'm thinking of redoing the git disect but this time really powercycle
the unit between kernels since it "seems/feels" like a timer which 
really counts to 0 and then locks the machine. 
After a lockup/freeze i cant boot the kernel another time with acpi 
enabled: it will simply hang after booting init.
And again nobody home: no keyboard activity whatsoever.

I've put the dmesg of 2.6.23-rc1-git6 with and without acpi=off online:
http://www.dth.net/kernel/via_output_2.6.23-rc1-git6-acpi_off
http://www.dth.net/kernel/via_output_2.6.23-rc1-git6-acpi_on

I would like to hear if someone has an idea how to tackle this problem.

Danny
-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Len Brown]

On Sunday 29 July 2007 22:54, Danny ter Haar wrote:
> Quoting Gabriel C (nix.or.die@googlemail.com):
> > Now while we think is ACPI this should be easy for you to bisect.
> > This commit http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=39804b20f62532fa05c2a8c3e2d1ae551fd0327b
> > merged ACPI so this one should be your first bad one.
> > Maybe Len has some idea and you don't need to bisect :)
> 
> Thanks to personal coaching of Gabriel i bisected the last few days.
> 
> It looked like this was the cullprit:
> 
> 22aadf8a07067644e101267ed5003043f2ad05bf is first bad commit

Please attach the output from acpidump to 
http://bugzilla.kernel.org/show_bug.cgi?id=7880
We'll likely be able to tell from it if that patch
has any real effect on your system, or if you test
result was from some unrelated event.

Also, please test with CONFIG_X86_ACPI_CPUFREQ=n
to remove the acpi-cpufreq driver (and thus this patch)
from your kernel.  If it still fails, then we know
that this driver (and this patch) are not related
to the failure.

> 2.6.23-rc1-git[1-6] all lockup solid after either direct or
> within a couple of minutes (less than 2) after reboot.
> 
> They all run fine with "acpi=off" as boot argument.

Hmmm, okay, the "big hammer" works.  Please see
if any of these smaller hammers work:

pnpacpi=off
acpi=noirq
notsc

thanks,
-Len

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Please attach the output from acpidump to 
> http://bugzilla.kernel.org/show_bug.cgi?id=7880

done
also on : http://www.dth.net/kernel/acpidump_output_via_c3_5000

> We'll likely be able to tell from it if that patch
> has any real effect on your system, or if you test
> result was from some unrelated event.

Reverting the patch didn't make a real difference, it only took
longer (eg 16 minutes before lockup in stead of 2-3 minutes)

> Also, please test with CONFIG_X86_ACPI_CPUFREQ=n
> to remove the acpi-cpufreq driver (and thus this patch)
> from your kernel.  If it still fails, then we know
> that this driver (and this patch) are not related
> to the failure.

Will do.
This machine(firewall) is solar powered so i tried to let it
consume as less power as possible 
This is in my startup script at the moment:

echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> 
> pnpacpi=off
> acpi=noirq
> notsc

More testing to do, however, sunday i'm going abroad for a while,
all stuff will be in storage. If i cant find in the next few days
(packing everything in moving boxes) it might take a while before
i can test again. Sorry for that.

Danny

-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> 
> pnpacpi=off
Went out for dinner, machine was frozen on return.
One less on the checklist ..

-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> acpi=noirq

died within 2 minutes

> notsc

After 36 minutes

-- 

Re: Update: ide problems: 2.6.22-git17 working, 2.6.23-rc1* is not:

[Danny ter Haar]

Quoting Len Brown (lenb@kernel.org):
> Also, please test with CONFIG_X86_ACPI_CPUFREQ=n
> to remove the acpi-cpufreq driver (and thus this patch)
> from your kernel.  If it still fails, then we know
> that this driver (and this patch) are not related
> to the failure.

It wasn't enabled in any of the kernels i ran:

# grep X86_ACPI ../configs/config*
../configs/config-2.6.22-git14-c3-via5000-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.22-git17-c3-via5000-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git5:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git6:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-git9:# CONFIG_X86_ACPI_CPUFREQ is not set
../configs/config-2.6.23-rc1-mm1-c3-firewall:# CONFIG_X86_ACPI_CPUFREQ is not set

> Hmmm, okay, the "big hammer" works.  Please see
> if any of these smaller hammers work:
> 
> pnpacpi=off
> acpi=noirq
> notsc

_none_ of these options will work. The machine locks up solid.
And that's what i'm the most surprised about. The last time i withnessed
a solid kernel lockup without any panic/notification is _years_ ago.

I also tried a kernel without "CPU Frequency scaling", so it is not the
scaling up/down in Mhz of the cpu.
I've even just compiled/run a kernel with libata and even that one still 
locks up.

New round of git bisects i guess (this time powercycle instead of reboot
and wait few hours before ginving it "bisect good" )

Will report back.

Danny

-- 

Re: Update: ACPI problems: 2.6.22-git17 working, 2.6.23-rc1* is not: (was ide not working)

[Danny ter Haar]

Okay

Redid git bisect from start:

git bisect log
git-bisect start
# good: [098fd16f00005f665d3baa7e682d8cb3d7c0fe6f] Linux 2.6.22
git-bisect good 098fd16f00005f665d3baa7e682d8cb3d7c0fe6f
# good: [098fd16f00005f665d3baa7e682d8cb3d7c0fe6f] Linux 2.6.22
git-bisect good 098fd16f00005f665d3baa7e682d8cb3d7c0fe6f
# bad: [7d57c74238cdf570bca20b711b2c0b31a553c1e5] Linux 2.6.23-rc1
git-bisect bad 7d57c74238cdf570bca20b711b2c0b31a553c1e5
# good: [1f1c2881f673671539b25686df463518d69c4649] Merge branch
# 'upstream-linus' of
# master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/netdev-2.6
git-bisect good 1f1c2881f673671539b25686df463518d69c4649
# good: [5b7f13bd26a0c1d394a1a1f2bb6de5130c3a3843] coda: update module
# information
git-bisect good 5b7f13bd26a0c1d394a1a1f2bb6de5130c3a3843
# good: [74889e41d9a1f80928130a02af9b010673bc5ba7] [CELL] cbe_cpufreq:
# reorganize code
git-bisect good 74889e41d9a1f80928130a02af9b010673bc5ba7
# good: [bdda1561ffda764583a295229db66d94cf6038a3] x86_64: x86_64 - Use
# non locked version for local_cmpxchg()
git-bisect good bdda1561ffda764583a295229db66d94cf6038a3
# bad: [d7f5e3df3574c6e38b99f5fe22f15540b2b9811d] Merge branch
# 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds
git-bisect bad d7f5e3df3574c6e38b99f5fe22f15540b2b9811d
# bad: [08e31686d6d119ba26bf0690f5f872f6f5bd1a97] Pull thinkpad into
# release branch
git-bisect bad 08e31686d6d119ba26bf0690f5f872f6f5bd1a97
# bad: [f79e3185dd0f8650022518d7624c876d8929061b] Pull misc into release
# branch
git-bisect bad f79e3185dd0f8650022518d7624c876d8929061b
# good: [22aadf8a07067644e101267ed5003043f2ad05bf] Pull bugzilla-7880
# into release branch
git-bisect good 22aadf8a07067644e101267ed5003043f2ad05bf
# good: [e8b495fe09bc793ae26774e7b2667f7f658d56e2] Pull dock-bay into
# release branch
git-bisect good e8b495fe09bc793ae26774e7b2667f7f658d56e2
# good: [18eab8550397f1f3d4b8b2c5257c88dae25d58ed] ACPI: Enable C3 even
# when PM2_control is zero
git-bisect good 18eab8550397f1f3d4b8b2c5257c88dae25d58ed
# good: [7a9b9068b8c7b526894904c0b9b90354825b8fbd] ACPI: update
# feature-removal-schedule.txt, /sys/firmware/acpi/namespace is gone
git-bisect good 7a9b9068b8c7b526894904c0b9b90354825b8fbd
# good: [8b8eb7d8cfc6cd95ed00cd58754e8493322505bd] ACPI: update ACPI
# proc I/F removal schedule
git-bisect good 8b8eb7d8cfc6cd95ed00cd58754e8493322505bd


f79e3185dd0f8650022518d7624c876d8929061b is first bad commit

I will store this particular firewall setup friday (will go into storage
for 2 months), if somebody would like more info, me to try something,
please send it to me asap.

Thanks for looking at this.

Danny
--