Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26 (LOW PRIORITY help request)

Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26 (LOW PRIORITY help request)

[trouble daemon]

Kernel hackers,

I consider this to be of rather low priority given the age of the
hardware, so ignore unless you feel like helping a fellow out, thanks!

I have a pair of Dell PowerEdge 4200's (dual 300 SMP, 512mb, SCA
hotswap scsi, have AMI MegaRaid but removed, same problem with or
without). I am using Debian Lenny, but have used etch (and in fact
bootstrap via FAI from etch currently). The 2.6.18 kernel that comes
with etch works fine in etch and Lenny, but when I tried to use the
2.6.26 kernel in Lenny, the kernel starts to panic (see "dmesg pastes"
below for output).

Ultimately, I was wondering if anyone might be able to take a look at
my dmesg's that might have any idea as to what feature might be new in
.26 that could be responsible for the panic. Sadly, it takes about 5
hours to do a clean run of make-kpkg in debian on these machines, and
I am not setup for distcc or cross compile on my x64 machine yet, so
it is a painful experience to recompile over and over to try figure
out what kernel feature it might be that is causing this.

I should note that I have tried one recompile of 2.6.26 (debianized)
to disable the new aic7xxx module and only use the old driver, but to
no avail. I am looking to try the 2.6.34 vanilla kernel next however.


dmesg pastes
---------------------
I have uploaded the panic dmesg to pastebin at (never expires):
http://pastebin.ca/1886636
I have uploaded the working 2.6.18 dmesg to pastbin at (never
expires): http://pastebin.ca/1886641


I think that about does it as far as pastebins go, but feel free to
ask for more information if you think you might have any idea. Since I
am not an LKML subscriber yet, I would please ask that any replies be
sent to this email address directly. Much appreciated for any help in
advance, thanks!



troubled

Re: Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26 (LOW PRIORITY help request)

[Konrad Rzeszutek Wilk]

> Ultimately, I was wondering if anyone might be able to take a look at
> my dmesg's that might have any idea as to what feature might be new in
> .26 that could be responsible for the panic. Sadly, it takes about 5
> hours to do a clean run of make-kpkg in debian on these machines, and
> I am not setup for distcc or cross compile on my x64 machine yet, so
> it is a painful experience to recompile over and over to try figure
> out what kernel feature it might be that is causing this.

It looks as the fdc driver is stuck in a loop and the init process never
gets pass this stage.

To see if that is the culprit, find the floppy.ko and rename it to say
floppy.ko.bak. Then recreate the initramfs.

The more worrying is the irqs firring. Try to use the 'irqpoll'
parameter as it suggests.

Re: Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26 (LOW PRIORITY help request)

[trouble daemon]

Konrad,

> The more worrying is the irqs firring. Try to use the 'irqpoll' parameter as
> it suggests.

Actually, it seems that it was solved with "noapic" kernel option, as was
suggested by Ben in the debian bug report at:

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=586494

I do thank you for replying though. I am still not sure what exactly changed in
debian between epoch (</joke>) and 2.6.26 though, since the dell's
have been 100% perfect
out of the box in every single debian since circa 2002, when I got them.

I can deal with noapic, but I will try the irqpoll as well. I only noticed that
the dmesg was mentioning it after the noapic solved my problem. One thing I did
notice with noapic though, is that network ping's seem to round down, so I see
lots of "0.000 ms" pings on the local network. I assume that without apic that
the system is just using some sort of timer to poll the nics though.

On a side note, I also saw a 2007 post from Andrew Morton, I think, that was
complaining about noapic and trying to track down where the problem might have
started. Given the approx. 1999 munfacturing date (from what I can tell) of the
PowerEdge 4200 series, and a perfect linux history from at least debian potato
up until 2.6.26, you can bet some money that someone, somewhere out there,
changed yet another feature that required me to use noapic. More specifically,
it worked in 2.6.18 in debian etch/lenny, but broke in the next debian kernel,
2.6.26. ie: the change/problem lies somewhere between 19-26 inclusive
(depending on if my apic issue isn't simply a debian default/tweak
that messed things
up, of course)

Hopefully Andrew high lights on his name! :)

Anyways, thanks and sorry for the wall of text! l8r



troubled

Re: Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26

[Jonathan Nieder]

(resending with a newer address for Alexey - sorry for the noise)
Hi,

trouble daemon wrote:

> I am using a pair of Dell PowerEdg 4200's that have been working with
> debian since 2002 or so, 100% perfectly.
[...]
>   - 2.6.26 kernel that panics (latest unmodified debian kernel):
> http://pastebin.ca/1886636
>   - 2.6.18 kernel that works (latest unmodified debian kernel):
> http://pastebin.ca/1886641

pastebin.ca doesn't seem to exist any more, alas.

[...]
> Ben Hutchings wrote:

>> Another option that may be worth trying is 'noapic'.
>
> Wow, that worked just fine actually on 2.6.26 stock kernel. Thanks a lot!
[...]
> f8924e770e048429ae13bfabe1ddad9bf1e64df7 is first bad commit
> commit f8924e770e048429ae13bfabe1ddad9bf1e64df7
> Author: Alexey Starikovskiy <astarikovskiy@suse.de>
> Date:   Fri Apr 4 23:42:21 2008 +0400
>
>    x86: unify mp_bus_info
[...]
> I have verified that checking out the patch just before this one works
> fine on the machines.

Sorry we dropped the ball on this.  Dan, any news (e.g., do later
kernels do any better)?  Alexey, any ideas for tracking this down?

Thanks,
Jonathan

Re: Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26

[trouble daemon]

Hello,

On Mon, Sep 12, 2011 at 2:56 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Sorry we dropped the ball on this.  Dan, any news (e.g., do later
> kernels do any better)?  Alexey, any ideas for tracking this down?

No worries, not like these machines are useful for much beyond museum
pieces any more ;) Sorry about the pastebin.ca though, I didn't
realize that they were gone.

As for later kernels, I have no idea tbh. Currently they seem to run
flawless with noapic, so that is what I use when installing and
running them. I do know that the Debian Squeeze installer chokes if I
don't use noapic, however.

If you are interested in some closure, I suppose I could take a shot
at installing something a little more recent and see if I can get away
with removing noapic though. Just let me know what kernel source
(vanilla/deb sid/etc), version, or any particular .config options that
you would like me to test, and I can see about pasting the console
terminal output when running without noapic disabled. I won't use a
pastebin this time, I promise ;)

Re: Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26

[Jonathan Nieder]

trouble daemon wrote:

> As for later kernels, I have no idea tbh. Currently they seem to run
> flawless with noapic, so that is what I use when installing and
> running them. I do know that the Debian Squeeze installer chokes if I
> don't use noapic, however.
>
> If you are interested in some closure, I suppose I could take a shot
> at installing something a little more recent and see if I can get away
> with removing noapic though. Just let me know what kernel source
> (vanilla/deb sid/etc), version, or any particular .config options that
> you would like me to test, and I can see about pasting the console
> terminal output when running without noapic disabled.

Thanks!  Really, anything 3.0 or later would be interesting (mainline
is better than distro-patched), with a .config that panics in an older
version.  Testing with a stock Debian squeeze kernel would be
interesting, too.

Re: Dell PowerEdge 4200 scsi, worked in 2.6.18, not in 2.6.26

[trouble daemon]

Heya,

On Fri, Sep 16, 2011 at 5:48 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> trouble daemon wrote:
>
>> As for later kernels, I have no idea tbh. Currently they seem to run
>> flawless with noapic, so that is what I use when installing and
>> running them. I do know that the Debian Squeeze installer chokes if I
>> don't use noapic, however.
>>
>> If you are interested in some closure, I suppose I could take a shot
>> at installing something a little more recent and see if I can get away
>> with removing noapic though. Just let me know what kernel source
>> (vanilla/deb sid/etc), version, or any particular .config options that
>> you would like me to test, and I can see about pasting the console
>> terminal output when running without noapic disabled.
>
> Thanks!  Really, anything 3.0 or later would be interesting (mainline
> is better than distro-patched), with a .config that panics in an older
> version.  Testing with a stock Debian squeeze kernel would be
> interesting, too.
>

Well, I grabbed a git copy from about 2 days ago and compiled/booted
it up. It was 3.1.0-rc7, and with noapic, it boots up debian squeeze
just fine, and if I leave noapic option out of grub, it fails to boot.
It wasn't a panic though.

Basically, it got to the point where it tried to load its megaraid and
aic7xx modules, and then choked when it hit cryptsetup, since the
drives weren't available. It was complaining about something to do
with INTD and INTA not being available or something. I actually
recompiled an extra time without aic7xx new driver (used old) to see
if it helped since the first time aic7xx was complaining about some
kind of scsi reset over and over, but it didn't help. On the
recompile, it just started happening to the megaraid instead.

Sorry that I don't have the dmesg for you just yet. I need to minicom
over and capture, save, etc. to get that off that machine. Just
thought I would let you know that noapic is still required, before you
forget about me :)

Anyways, let me know if you even need the bootup log or not still.
Talk to you soon. \o