2.4.5-ac12 kernel oops

2.4.5-ac12 kernel oops

[Colonel]

Why the symbol mismatch?  Why ignore /proc over the System.map?

The system had very little running (because of prior lockups) during
the morning SQL activity. About 3/4 of a megabyte was being added to
the SQL databases, and that activity had been going on for 45 minutes
prior to the oops.  No X, and I was reading email as the only other
load.  Daemons: postfix, raid5, xntpd, syslog, cron & kernel.  SMP
kernel, built with gcc 2.95.3 released 20010315.

At the time of the oops, there is an cron task that feeds postfix,
which deposits the email into a 3 meg file.



------------------------------------------------------------------

ksymoops 2.4.1 on i686 2.4.5-ac12.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.5-ac12/ (default)
     -m /System.map-2.4.5-ac12 (specified)

Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base says c01aad00, System.map says c014cba0.  Ignoring ksyms_base entry
Unable to handle kernel NULL pointer dereference at virtual address 00000015
c0147e62
*pde = 00000000
Oops: 0002
CPU:    0
EIP:    0010:[<c0147e62>]
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010206
eax: 00000005   ebx: c6ef1800   ecx: c021ed64   edx: c2910000
esi: c021ee80   edi: 0000000b   ebp: c5e2f340   esp: c601fea4
ds: 0018   es: 0018   ss: 0018
Process ps (pid: 10454, stackpage=c601f000)
Stack: c0144eae c6ef1800 c2910000 c7ff6000 c0148ffd c6ef1800 c021f1c0 c5e2f3a4 
       c01f79b8 c0149246 c7ff6000 c2910000 0000000b fffffff4 c1e85220 c601e000 
       c5e2f340 ffffffea c013abae c1e85220 c5e2f340 c601ff34 00000000 c1e85220 
Call Trace: [<c0144eae>] [<c0148ffd>] [<c0149246>] [<c013abae>] [<c013b2eb>] 
   [<c013bb4e>] [<c014392a>] [<c012fe23>] [<c013013e>] [<c0106c63>] 
Code: f0 ff 48 10 8b 42 24 80 48 14 08 52 e8 0d ff ff ff 83 c4 04 

>>EIP; c0147e62 <proc_delete_inode+32/48>   <=====
Trace; c0144eae <iput+ba/178>
Trace; c0148ffd <proc_pid_make_inode+ad/b8>
Trace; c0149246 <proc_base_lookup+86/23c>
Trace; c013abae <real_lookup+7a/108>
Trace; c013b2eb <path_walk+58f/7c4>
Trace; c013bb4e <open_namei+86/598>
Trace; c014392a <destroy_inode+1a/20>
Trace; c012fe23 <filp_open+3b/5c>
Trace; c013013e <sys_open+36/cc>
Trace; c0106c63 <system_call+33/38>
Code;  c0147e62 <proc_delete_inode+32/48>
0000000000000000 <_EIP>:
Code;  c0147e62 <proc_delete_inode+32/48>   <=====
   0:   f0 ff 48 10               lock decl 0x10(%eax)   <=====
Code;  c0147e66 <proc_delete_inode+36/48>
   4:   8b 42 24                  mov    0x24(%edx),%eax
Code;  c0147e69 <proc_delete_inode+39/48>
   7:   80 48 14 08               orb    $0x8,0x14(%eax)
Code;  c0147e6d <proc_delete_inode+3d/48>
   b:   52                        push   %edx
Code;  c0147e6e <proc_delete_inode+3e/48>
   c:   e8 0d ff ff ff            call   ffffff1e <_EIP+0xffffff1e> c0147d80 <de_put+0/b0>
Code;  c0147e73 <proc_delete_inode+43/48>
  11:   83 c4 04                  add    $0x4,%esp


1 warning issued.  Results may not be reliable.

Re: 2.4.5-ac12 kernel oops

[Keith Owens]

On Mon, 25 Jun 2001 08:15:49 -0700 (PDT), 
klink@clouddancer.com (Colonel) wrote:
>ksymoops 2.4.1 on i686 2.4.5-ac12.  Options used
>Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base says c01aad00, System.map says c014cba0.  Ignoring ksyms_base entry
>Why the symbol mismatch?

The mismatch is caused by two variables called partition_name.  What
does 'nm vmlinux | grep partition_name' show?  Probably one
partition_name at c01aad00 and another at c014cba0.  Both
fs/partitions/msdos.c and drivers/md/md.c define that symbol, md
exports its version.  A good reason why exported symbols should have
unique names.

>Why ignore /proc over the System.map?

ksymoops has a hierarchy of trust.  System.map is more trustworthy than
/proc/ksyms because ksyms changes, especially if you rebooted after the
oops and before running ksymoops.

Re: 2.4.5-ac12 kernel oops

[Colonel]

In clouddancer.list.kernel, you wrote:
>
>On Mon, 25 Jun 2001 08:15:49 -0700 (PDT), 
>klink@clouddancer.com (Colonel) wrote:
>>ksymoops 2.4.1 on i686 2.4.5-ac12.  Options used
>>Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base says c01aad00, System.map says c014cba0.  Ignoring ksyms_base entry
>>Why the symbol mismatch?
>
>The mismatch is caused by two variables called partition_name.  What
>does 'nm vmlinux | grep partition_name' show?  Probably one

Can't run that, but : 

grep "partition_name" System.map-2.4.5-ac12 
00000000c014cba0 t partition_name
00000000c01aad00 T partition_name


>partition_name at c01aad00 and another at c014cba0.  Both
>fs/partitions/msdos.c and drivers/md/md.c define that symbol, md

and I have both in the kernel.

>exports its version.  A good reason why exported symbols should have
>unique names.

Only the raid5 was active, the msdos stuff is a module for 'just in
case'.  Something else triggered this, I've run raid for a long time
without problems.



>>Why ignore /proc over the System.map?
>
>ksymoops has a hierarchy of trust.  System.map is more trustworthy than
>/proc/ksyms because ksyms changes, especially if you rebooted after the
>oops and before running ksymoops.

Hmm, I would have thought that /proc was more up to date, because it
would reflect changes.  No reboot, never even considered it (I've
rescued too many junior sysadmins that think rebooting is _the_ answer).




-- 
"Or heck, let's just make the VM a _real_ Neural Network, that self
trains itself to the load you put on the system. Hideously complex and
evil?  Well, why not wire up that roach on the floor, eating that stale
cheese doodle. It can't do any worse job on VM that some of the VM
patches I've seen..."  -- Jason McMullan

ditto

Re: 2.4.5-ac12 kernel oops

[Keith Owens]

On Mon, 25 Jun 2001 09:02:19 -0700 (PDT), 
klink@clouddancer.com (Colonel) wrote:
>In clouddancer.list.kernel, Keith Owens wrote:
>>partition_name at c01aad00 and another at c014cba0.  Both
>>fs/partitions/msdos.c and drivers/md/md.c define that symbol, md
>
>and I have both in the kernel.
>Only the raid5 was active, the msdos stuff is a module for 'just in
>case'.  Something else triggered this, I've run raid for a long time
>without problems.

Both msdos and raid are in your kernel, msdos is not a module.

Don't be misled by ksymoops reporting duplicate symbols.  It reads the
entire symbol map and reports discrepancies as it does so, mainly to
pick up idiots who feed /proc/ksyms from 2.2 and System.map from 2.4.
Don't laugh, it happens!  Unless the trace shows raid or msdos are
implicated in the oops, you can ignore the ksymoops warning.

>>ksymoops has a hierarchy of trust.  System.map is more trustworthy than
>>/proc/ksyms because ksyms changes, especially if you rebooted after the
>>oops and before running ksymoops.
>
>Hmm, I would have thought that /proc was more up to date, because it
>would reflect changes.  No reboot, never even considered it (I've
>rescued too many junior sysadmins that think rebooting is _the_ answer).

/proc/ksyms is dynamic, it changes as modules are loaded and unloaded.
And often the oops is so bad that the machine is dead so reboot is the
only option, ksyms after reboot may be for a completely different
kernel.  If you want accurate ksyms and lsmod data to feed into
ksymoops, 'man insmod' and read section 'KSYMOOPS ASSISTANCE'.

Re: 2.4.5-ac12 kernel oops

[Colonel]

>>Hmm, I would have thought that /proc was more up to date, because it
>>would reflect changes.  No reboot, never even considered it (I've
>>rescued too many junior sysadmins that think rebooting is _the_ answer).
>
>/proc/ksyms is dynamic, it changes as modules are loaded and unloaded.
>And often the oops is so bad that the machine is dead so reboot is the
>only option, ksyms after reboot may be for a completely different
>kernel.  If you want accurate ksyms and lsmod data to feed into
>ksymoops, 'man insmod' and read section 'KSYMOOPS ASSISTANCE'.

OK, I added the cron task and directory, perhaps I need them sometime.
In any event, thanks for the info.  This certainly seems very well
thought out!


-- 
"Or heck, let's just make the VM a _real_ Neural Network, that self
trains itself to the load you put on the system. Hideously complex and
evil?  Well, why not wire up that roach on the floor, eating that stale
cheese doodle. It can't do any worse job on VM that some of the VM
patches I've seen..."  -- Jason McMullan

ditto