Kernel Panic in 2.2.x

Kernel Panic in 2.2.x

[Hawkins Jeffrey-CJH016]

Request for Info/Feedbak....

With a Standard 2.2.17 Kernel, with some Proprietary Hardware Drivers,
we intermittently encounter a Kernel Panic due to Reference to a NULL
Pointer.  I have isolated the NULL Reference to the "procfs" Support.
In particular, in "array.c", the "get_stat" function, with usage
of the KSTK_EIP and KSTK_ESP Macros.  The NULL access is due to
the "regs" pointer in the "tss" structure being NULL.  My theory
is there is a race condition with procfs access and a process
terminating at the same time.  At the time of a our failure, a
Process is terminating (a Daemon Restart induced by our Application),
as well as,  one of our Application's is performing Raw Socket
I/O for Network Monitoring -- the strange thing is that if we
remove the Raw Socket Functionality we can not get the Failure
to occur.

I noticed in the 2.4.x Tree the KSTK_ Macros have been modified
to check for NULL. Does anybody know if this was the reason for
the change.  Looking at the Kernel List Archives, it seems the
change was for "init" issues in "BootX"?

Also, reviewing the Kernel List Archives, I noticed in 2.2.x
there was a race condition with "procfs" access, but related
to the MM Stats/Params of a Process, not the TSS Registers.

Anybody have any insight into this Issue?

Also, insight into how the tss->regs is utilized and updated
would be appreciated.  I have started reviewing the PPC Specific
Kernel Code to get this info on the Task Switching Implementation,
but I thought maybe someone here could give me some insight, or
direct me to a Book/URL/Reference that has this type of information.

With respect to responses, please don't say go to the 2.4.x Kernel
as a solution for the Issue....:)  This is in our plans, but at this time,
we are locked into the 2.2 Kernel due to Proprietary Hardware Driver
Support.  For the short term, I just want to identify the true root
cause (to appease the Management Gods), and to possibly implement
a short term fix until we migrate to the 2.4.x or 2.6 Kernel.


Jeff

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

RE: Kernel Panic in 2.2.x

[Hawkins Jeffrey-CJH016]

Follow-up to my Kernel Panic Investigations, it appears
that the Process as having "tss->regs" as NULL, during
the execution of "ps" command, is a modprobe being performed
by the Kernel for attempting to Load net-pf-10 Module
(IPV6 Packet Filter).






> -----Original Message-----
> From: Hawkins Jeffrey-CJH016
> Sent: Thursday, May 29, 2003 2:05 PM
> To: linuxppc-dev@lists.linuxppc.org
> Subject: Kernel Panic in 2.2.x
>
>
>
> Request for Info/Feedbak....
>
> With a Standard 2.2.17 Kernel, with some Proprietary Hardware Drivers,
> we intermittently encounter a Kernel Panic due to Reference to a NULL
> Pointer.  I have isolated the NULL Reference to the "procfs" Support.
> In particular, in "array.c", the "get_stat" function, with usage
> of the KSTK_EIP and KSTK_ESP Macros.  The NULL access is due to
> the "regs" pointer in the "tss" structure being NULL.  My theory
> is there is a race condition with procfs access and a process
> terminating at the same time.  At the time of a our failure, a
> Process is terminating (a Daemon Restart induced by our Application),
> as well as,  one of our Application's is performing Raw Socket
> I/O for Network Monitoring -- the strange thing is that if we
> remove the Raw Socket Functionality we can not get the Failure
> to occur.
>
> I noticed in the 2.4.x Tree the KSTK_ Macros have been modified
> to check for NULL. Does anybody know if this was the reason for
> the change.  Looking at the Kernel List Archives, it seems the
> change was for "init" issues in "BootX"?
>
> Also, reviewing the Kernel List Archives, I noticed in 2.2.x
> there was a race condition with "procfs" access, but related
> to the MM Stats/Params of a Process, not the TSS Registers.
>
> Anybody have any insight into this Issue?
>
> Also, insight into how the tss->regs is utilized and updated
> would be appreciated.  I have started reviewing the PPC Specific
> Kernel Code to get this info on the Task Switching Implementation,
> but I thought maybe someone here could give me some insight, or
> direct me to a Book/URL/Reference that has this type of information.
>
> With respect to responses, please don't say go to the 2.4.x Kernel
> as a solution for the Issue....:)  This is in our plans, but
> at this time,
> we are locked into the 2.2 Kernel due to Proprietary Hardware Driver
> Support.  For the short term, I just want to identify the true root
> cause (to appease the Management Gods), and to possibly implement
> a short term fix until we migrate to the 2.4.x or 2.6 Kernel.
>
>
> Jeff
>
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[Hollis Blanchard]

On Friday, May 30, 2003, at 11:11 US/Central, Hawkins Jeffrey-CJH016
wrote:
>>
>> With respect to responses, please don't say go to the 2.4.x Kernel
>> as a solution for the Issue....:)  This is in our plans, but
>> at this time,
>> we are locked into the 2.2 Kernel due to Proprietary Hardware Driver
>> Support.  For the short term, I just want to identify the true root
>> cause (to appease the Management Gods), and to possibly implement
>> a short term fix until we migrate to the 2.4.x or 2.6 Kernel.

Hi Jeff, I know it's important to you, but I would be pretty surprised
if you could interest anyone else with a 2.2 bug. :) You may want to
bring that to your management's attention: by continuing with 2.2,
you'll get little community support, which would otherwise be quite
valuable in situations exactly like this one.

--
Hollis Blanchard
IBM Linux Technology Center


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[linas]

On Fri, May 30, 2003 at 11:20:50AM -0500, Hollis Blanchard wrote:
>
> On Friday, May 30, 2003, at 11:11 US/Central, Hawkins Jeffrey-CJH016
> wrote:
> >>
> >> With respect to responses, please don't say go to the 2.4.x Kernel
> >> as a solution for the Issue....:)  This is in our plans, but
> >> at this time,
> >> we are locked into the 2.2 Kernel due to Proprietary Hardware Driver
> >> Support.  For the short term, I just want to identify the true root
> >> cause (to appease the Management Gods), and to possibly implement
> >> a short term fix until we migrate to the 2.4.x or 2.6 Kernel.
>
> Hi Jeff, I know it's important to you, but I would be pretty surprised
> if you could interest anyone else with a 2.2 bug. :) You may want to

I did read in slashdot about some group that was planing on supporting
version 2.2 indefinitely.

Its not a bad idea ... I run a 2.2 kernel on a server; I haven't switched
to 2.4 because the particular tool that I need was never ported to 2.4.
I occasionally think about doing the port myself, but time constraints
& all ... I'm expecting to keep that server on 2.2 forever, or at least
till the hardware dies.

Latest & greatest is only the best if you are a developer.  For users,
old-trustworthy is usually a much better bet.

--linas

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[Hollis Blanchard]

On Friday, May 30, 2003, at 11:34 US/Central, linas@austin.ibm.com
wrote:
>
> Latest & greatest is only the best if you are a developer.  For users,
> old-trustworthy is usually a much better bet.

I disagree, but even if that is the case, what Jeff needs is clearly
developer support... :)

--
Hollis Blanchard
IBM Linux Technology Center


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[linas]

On Fri, May 30, 2003 at 11:40:49AM -0500, Hollis Blanchard wrote:
> On Friday, May 30, 2003, at 11:34 US/Central, linas@austin.ibm.com
> wrote:
> >
> > Latest & greatest is only the best if you are a developer.  For users,
> > old-trustworthy is usually a much better bet.
>
> I disagree,

OK, I can't just let this one slide.

It's hard to say this without sounding personal and insulting, but it
is exactly this sort of attitude on the part of developers that lies
at the root of a lot of the software attrocities commited (of which
I am a guilty, collaborating member).  This sort of an attitude has
got to change; if not, people will curse Linux as much as they curse
Windows.

At various points in my life (including the present) I have had to
sysadmin more than half-a-dozen systems at the same time, and software
upgrades are one of the worst things a sysadmin has to go through.
Things like disk crashes and data loss are pretty bad,  but software
upgrades often result in data loss too, or system outages, or subsystem
outages.  No sysadmin willingly upgrades a working system.

I could barrage you with a litany of personal experiences.  I could
tell you to talk to any sysadmin, who would do the same.   I could
try to remind you of what you went through last time you accidentally
got hammered by an MS Windows upgrade.  Or a debian apt-get upgrade.

I have spent way too much of my life trying to deal with brokeness,
and frankly, I'm kinda getting sick of it.   I'm feeling gun-shy or
shell-shocked or whatever, I get this sinking feeling, 'batten down
the hatches', 'get ready for another all-nighter', 'there goes another
nice sunny weekend' every time I have to contemplate a software upgrade.

Yes, I am venting, but there was a time when I enjoyed maintaining
my own Linux server. Now I hate it, and my hatred is reaching crisis
proportions.  I know that I am not alone.  OK, so what to do?  Can
I steer the conversation back to something positive & constructuive?

I've started writing up some of these problems here:

http://www.linas.org/linux/peeves.html

This url deals more with a hardware outage I had running www.gnucash.org,
(you can read a short version of the saga on the news, there), and
deals primarily with storage issues.   I'll expand this tirade to
include software upgrades: I think I can blame at least some of my
pain on an upgrade of the 'mailman' mailing list software, which
clobbered all of the subscriber lists. I had to downgrade back to the
older version of the software, and restore what I could from backup.

Ugh.

--linas

Hmm. Its tempting to blame the upgrade process, rather than the new
software.  The upgrade process stinks, and needs a complete overhaul.
But sometimes the mere act of upgrading is at fault.  I remember installing
kernel 2.2.13 (or some version like that) and the mouse just stoped
working for no apparent reason.  Even a picture-perfect upgrade
process would still let things like that sneak through: the moral
of the story being that software upgrades are inherently risky.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[Hollis Blanchard]

On Friday, May 30, 2003, at 16:48 US/Central, linas@austin.ibm.com
wrote:
>
> On Fri, May 30, 2003 at 11:40:49AM -0500, Hollis Blanchard wrote:
>> On Friday, May 30, 2003, at 11:34 US/Central, linas@austin.ibm.com
>> wrote:
>>>
>>> Latest & greatest is only the best if you are a developer.  For
>>> users,
>>> old-trustworthy is usually a much better bet.
>>
>> I disagree,
>
> At various points in my life (including the present) I have had to
> sysadmin more than half-a-dozen systems at the same time, and software
> upgrades are one of the worst things a sysadmin has to go through.
> Things like disk crashes and data loss are pretty bad,  but software
> upgrades often result in data loss too, or system outages, or subsystem
> outages.  No sysadmin willingly upgrades a working system.

But you're venting entirely from a sysadmin standpoint. The position
you advocate here nothing new; in fact it's common practice. Even in
the Linux world: have a look at RHAS or Debian "stable" - these things
are constantly on the verge of obsolescence (and that's being kind,
depending on when the last Debian release was ;).

Now think about it from a general user standpoint. "Sleep crashes my
powerbook." "My Xserve doesn't boot." Video drivers, SMP stability and
performance... Even getting away from hardware support, personally I
*require* the input layer, which allows me to plug and unplug USB mice
while also using a PS/2 nipplemouse without freaking out my X server.
It didn't always work like that. Wheelie-mouse support - it wasn't
always there! And now let's step away from the kernel: I'm sure you're
a diehard twm user, but I think we can all agree the KDE and Gnome
projects of today are much better than those from two years ago. And
anyone here who remembers trying to compile x86-developed code with
PowerPC glibc 1.99 will assure you that newer is better. :)

For many (most?) users, newer is better. If that weren't the case, then
what are all these developers *doing* anyways? :) As for sysadmins,
you're free (wise, even :)to keep doing what you've always done: if it
ain't broke don't fix it.

--
Hollis Blanchard
IBM Linux Technology Center


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

RE: Kernel Panic in 2.2.x

[Hawkins Jeffrey-CJH016]

With respect to the Issue of the PANIC, I have made a
simple correction in the Kernel procfs Support.  The
correction/change was to add a NULL Check to the
KSTK_ Macros (this fix is already in the 2.4.x Kernel).

The root cause was as I indicated,  access to the /proc
file system by a "ps" command, and the execution of a
Module Load Attempt (via a modprobe) by the Kernel.  The
TSS Registers Pointer for the Process/Thread executing
the modprobe being NULL.

In anycase, I am happy with the small fix, and will be
testing it over the weekend with the failure scenario we
were able to produce the Panic.





> -----Original Message-----
> From: Hawkins Jeffrey-CJH016 [mailto:Jeffrey.F.Hawkins@Motorola.com]
> Sent: Friday, May 30, 2003 11:11 AM
> To: linuxppc-dev@lists.linuxppc.org
> Subject: RE: Kernel Panic in 2.2.x
>
>
>
> Follow-up to my Kernel Panic Investigations, it appears
> that the Process as having "tss->regs" as NULL, during
> the execution of "ps" command, is a modprobe being performed
> by the Kernel for attempting to Load net-pf-10 Module
> (IPV6 Packet Filter).
>
>
>
>
>
>
> > -----Original Message-----
> > From: Hawkins Jeffrey-CJH016
> > Sent: Thursday, May 29, 2003 2:05 PM
> > To: linuxppc-dev@lists.linuxppc.org
> > Subject: Kernel Panic in 2.2.x
> >
> >
> >
> > Request for Info/Feedbak....
> >
> > With a Standard 2.2.17 Kernel, with some Proprietary
> Hardware Drivers,
> > we intermittently encounter a Kernel Panic due to Reference
> to a NULL
> > Pointer.  I have isolated the NULL Reference to the
> "procfs" Support.
> > In particular, in "array.c", the "get_stat" function, with usage
> > of the KSTK_EIP and KSTK_ESP Macros.  The NULL access is due to
> > the "regs" pointer in the "tss" structure being NULL.  My theory
> > is there is a race condition with procfs access and a process
> > terminating at the same time.  At the time of a our failure, a
> > Process is terminating (a Daemon Restart induced by our
> Application),
> > as well as,  one of our Application's is performing Raw Socket
> > I/O for Network Monitoring -- the strange thing is that if we
> > remove the Raw Socket Functionality we can not get the Failure
> > to occur.
> >
> > I noticed in the 2.4.x Tree the KSTK_ Macros have been modified
> > to check for NULL. Does anybody know if this was the reason for
> > the change.  Looking at the Kernel List Archives, it seems the
> > change was for "init" issues in "BootX"?
> >
> > Also, reviewing the Kernel List Archives, I noticed in 2.2.x
> > there was a race condition with "procfs" access, but related
> > to the MM Stats/Params of a Process, not the TSS Registers.
> >
> > Anybody have any insight into this Issue?
> >
> > Also, insight into how the tss->regs is utilized and updated
> > would be appreciated.  I have started reviewing the PPC Specific
> > Kernel Code to get this info on the Task Switching Implementation,
> > but I thought maybe someone here could give me some insight, or
> > direct me to a Book/URL/Reference that has this type of information.
> >
> > With respect to responses, please don't say go to the 2.4.x Kernel
> > as a solution for the Issue....:)  This is in our plans, but
> > at this time,
> > we are locked into the 2.2 Kernel due to Proprietary Hardware Driver
> > Support.  For the short term, I just want to identify the true root
> > cause (to appease the Management Gods), and to possibly implement
> > a short term fix until we migrate to the 2.4.x or 2.6 Kernel.
> >
> >
> > Jeff
> >
> >
>
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[Linas Vepstas]

Hi Hollis,


I got your reply, but don't have it here in front of me, so I can't
quote it. But ...

On Fri, May 30, 2003 at 04:48:22PM -0500, linas@austin.ibm.com was heard to remark:
> On Fri, May 30, 2003 at 11:40:49AM -0500, Hollis Blanchard wrote:
> > On Friday, May 30, 2003, at 11:34 US/Central, linas@austin.ibm.com
> > wrote:
> > >
> > > Latest & greatest is only the best if you are a developer.  For users,
> > > old-trustworthy is usually a much better bet.
> >
> > I disagree,
>
> OK, I can't just let this one slide.

In your reply, you talk mostly of Linux desktop apps, and new hardware.
Lets not dismiss the problem by false analogy.

-- New hardware (e.g. laptop which won't go into sleep mode)  does
   require new software, drivers, etc. But that's OK, cause its new
   hardware, not an upgrade.  I have nothing against new software:
   I create a lot of it, and I use a lot of it.

-- You also mention desktop apps. Yes, the linux desktop has been
   moving rapidly, and it is reasonable to assume that some users want
   to upgrade old hardware to use the new desktops. (instead of
   buying new hardware that actually has things like USB on it).

   But beware of turning the home user into an inadvertant sysadmin.
   When your home user becomes the sysadmin, they will rapidly come
   to hate you.

   A little annecdote: I 'inherited' a late-model Win98 box.  I 'loaned'
   it to my 6 year old.  Who used it happily.  One day, we got a new
   game that required a new 'macromedia flash'.  OK, I'm a big man,
   'I can do this', I say to myself.  It pre-req'ed an newer version
   of apple's 'quicktime veiwer'.  This pre-req'ed a pile of Win98
   upgrades from Microsoft, about 25MB worth.  I think it must have
   been an automated Win98SE upgrade.  Took a while to download.  When
   done, the sound card didn't work.  My 6 year old was unhappy.  The
   next day, 10 drivers later, sound still doesn't work.  Every nite,
   I come home, my 6 year old hovers waiting to get the computer back.
   This goes on for 5 evenings in a row.  In furstration, I tell him,
   sorry, computer's ba-ba.  We have discussion about piggy-banks and
   cost of new computer, and how spare change isn't gonna solve it.
   Next week a freind offers to pirate a copy of WinXP for me, he
   says its pretty good, he likes it.  I almost go for it: one thought
   holds me back:  new software sometimes has a real problem with
   old hardware.  I might be jumping from frying pan into fire on such
   an upgrade. Sometime, a few weeks later, after we've given up on it,
   I'm toying around, install the sound driver one more time, one more
   reboot, magic! it starts working!  Oh, and the game that required
   the software update? It still didn't work.

   Moral of the story? Even desktop users can be turned into sysadmins.

   (BTW, I am the proud co-author of gnucash, which, if you read the
   trade-press, is in the top-10, if not ranked #1 in the category of
   'most difficult-to-install Linux desktop software'.  Of which I'm
   *not* proud of. But what can one do?  How else can one add new
   features? So I understand the other side of the coin pretty well.)

Linux is real strong in the server space, weak in the desktop.  Server
'users' are almost exclusively sysadmins.  They run systems for 5 years
or more.  (my www.linas.org is an 9 year old Dell 486.  www.gnucash.org
is 3.5 years old)  When a server goes down, hundreds of users are
affected.  (each of the www.gnucash.org mailing lists have 500+
subscribers, I move 7-10 GBytes a month.).  I think we need to make sure
that Linux in the server space benefits the sysadmin, and this means
very conservative design principles, and conservative installers
(something that really can backout the new version and replace old
version).   God help us all if one day Microsoft succeeds in building
a system that is easier to install/upgrade/manage than a Linux server.
Don't bite the hand that is currently feeding Linux.

--linas

--
pub  1024D/01045933 2001-02-01 Linas Vepstas (Labas!) <linas@linas.org>
PGP Key fingerprint = 8305 2521 6000 0B5E 8984  3F54 64A9 9A82 0104 5933

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[Hollis Blanchard]

On Friday, May 30, 2003, at 19:58 US/Central, Linas Vepstas wrote:
>
> Linux is real strong in the server space, weak in the desktop.  Server
> 'users' are almost exclusively sysadmins.  They run systems for 5 years
> or more.  (my www.linas.org is an 9 year old Dell 486.  www.gnucash.org
> is 3.5 years old)  When a server goes down, hundreds of users are
> affected.  (each of the www.gnucash.org mailing lists have 500+
> subscribers, I move 7-10 GBytes a month.).  I think we need to make
> sure
> that Linux in the server space benefits the sysadmin, and this means
> very conservative design principles, and conservative installers
> (something that really can backout the new version and replace old
> version).   God help us all if one day Microsoft succeeds in building
> a system that is easier to install/upgrade/manage than a Linux server.
> Don't bite the hand that is currently feeding Linux.

There's no problem here. Keep right on installing <distribution from
1996> until your hardware dies, and it won't bother me one bit.
Backport security fixes (or find others who have already), and ignore
reports about the latest wonderful features in the current version.
Your 500+ subscribers will thank you.

However, don't complain that upgrading your system can make things
break and ask me for sympathy. :) I for one would rather spend my time
making things work than making things backwards-compatible back to
version 0.3. I guess you're trying to convince me of the exact
opposite, but IMHO there's simply too much stuff left to get working.
Once it all works then we can talk about backwards-compatibility... ;)

--
Hollis Blanchard
IBM Linux Technology Center

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel Panic in 2.2.x

[linas]

On Mon, Jun 02, 2003 at 09:53:03AM -0500, Hollis Blanchard wrote:
>
> However, don't complain that upgrading your system can make things
> break and ask me for sympathy. :) I for one would rather spend my time
> making things work than making things backwards-compatible back to

Well, part of the problem is, of course, that things that used to
work in old versions sometimes don't work in newer versions.  No
one will argue that 'making things work' isn't a good intention,
but, by itself, it is not enough to make an effective process.

Everyone has good intentions, but accidents still happen.  You say
that you want to drive 100 mph, and you think I'm saying 'everyone
must drive 55 mph'.  That's not what I'm saying.  I'm trying to talk
about what it takes to drive 100 mph *safely*.

> version 0.3. I guess you're trying to convince me of the exact
> opposite, but IMHO there's simply too much stuff left to get working.
> Once it all works

Surely, you know that that day will never, ever come!  You do understand
that, don't you?

> then we can talk about

That's like saying "once we have eliminated all wars and killing,
only then shall we start to think about medicine and health care!"

> backwards-compatibility... ;)

I don't think you understood what I'm saying.  Its not about backwards
compatibility.  Backwards-compat can help, but its *NOT* what I'm
talking about.   Did I actually use the words 'backwards compatible'
anywhere in any of my series of notes on this topic?  I don't think so!

Part of what I'm talking about is technology, like having an installer
that allows you install the new version without trashing the old version,
so that the new version can be backed out if it doesn't work.  I think
AIX has an installer like this (I've never used it).  Linux doesn't.

Part of what I'm talking about is a change of attitude, such as the
idea that maintaining and patching the 2.2 kernel is wrong.  It's
not wrong, and there are some very good technical reasons for
maintaining this kernel/code base.

--linas


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Remote serial console through USB

[daRonin]

  I am trying to get remote serial console going through my Keyspan
USB to Serial adapter.

  The machine is single G4 tower, 1Ghz CPU, running 2.4.20-ben10.

  I appended these options to the kernel load time parameters:

  console=ttyUSB0,57600

  The Keyspan adapter works fine in minicom, and dmesg seems to show all
the modules loading. However I get the message:

"  Warning: unable to open an initial console.  "

  After doing some research I discovered that there is a patch for 2.5
kernel that would allow me to accomplish this task.

  Is there any way I can get this going in 2.4?

  Regards,

daRonin

========================================================
Certe, Toto, sentio nos in Kansate non iam adesse


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/