Kernel BUG 2.4.19-pre8-ac1 + preempt

Kernel BUG 2.4.19-pre8-ac1 + preempt

[Erik McKee]

Hello

This output...

kernel BUG at /usr/src/linux-2.2.13/include/linux/mm_inline.h:78!
invalid operand: 0000
CPU:    0
EIP:    0010:[<c012dd02>]    Not tainted
EFLAGS: 00210202
eax: 01000049   ebx: c10c4b2c   ecx: c024159c   edx: c10d6c04
esi: c10c4b10   edi: c01325a8   ebp: ffffe000   esp: c52bdde0
ds: 0018   es: 0018   ss: 0018
Process kfind (pid: 20927, stackpage=c52bd000)
Stack: c02414c0 c024159c 0000045c 0000003b c02415cc c52bc000 0000045c 00000000
       00000042 c012df1c c024159c 000001d2 00000006 000001d2 c0241780 000001ff
       00000000 00000002 c012e750 000001d2 c52bc000 c0241780 c012ee6a 000001d2
Call Trace: [<c012df1c>] [<c012e750>] [<c012ee6a>] [<c012f8d3>] [<c012f666>]
   [<c0126442>] [<c01264ba>] [<c0127b0b>] [<c0123b5d>] [<c0123e13>]
[<c01117ab>]
   [<c0111694>] [<c01a7b0e>] [<c01a802c>] [<c01abb58>] [<c0109adc>]
[<c010bd08>]
   [<c0109cc6>] [<c01088e4>] 

Code: 0f 0b 4e 00 20 30 21 c0 8d b6 00 00 00 00 8b 46 18 a8 80 74
 <3>kfind[20927] exited with preempt_count 1

Which I decoded and got this...

ksymoops 2.3.7 on i586 2.4.19-pre8-ac1.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.19-pre8-ac1/ (default)
     -m /boot/Sys-2.4.19-pre8-ac1 (specified)

Warning (compare_maps): ksyms_base symbol
vmalloc_to_page_R__ver_vmalloc_to_page not found in System.map.  Ignoring
ksyms_base entry
Warning (compare_maps): mismatch on symbol packet_socks_nr  , af_packet says
c8986244, /lib/modules/2.4.19-pre8-ac1/kernel/net/packet/af_packet.o says
c8986164.  Ignoring /lib/modules/2.4.19-pre8-ac1/kernel/net/packet/af_packet.o
entry
Warning (compare_maps): mismatch on symbol journal_enable_debug  , jbd says
c88bffe4, /lib/modules/2.4.19-pre8-ac1/kernel/fs/jbd/jbd.o says c88bffd0. 
Ignoring /lib/modules/2.4.19-pre8-ac1/kernel/fs/jbd/jbd.o entry
Warning (compare_maps): mismatch on symbol unix_socket_table  , unix says
c88b37a0, /lib/modules/2.4.19-pre8-ac1/kernel/net/unix/unix.o says c88b3540. 
Ignoring /lib/modules/2.4.19-pre8-ac1/kernel/net/unix/unix.o entry
kernel BUG at /usr/src/linux-2.2.13/include/linux/mm_inline.h:78!
invalid operand: 0000
CPU:    0
EIP:    0010:[<c012dd02>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210202
eax: 01000049   ebx: c10c4b2c   ecx: c024159c   edx: c10d6c04
esi: c10c4b10   edi: c01325a8   ebp: ffffe000   esp: c52bdde0
ds: 0018   es: 0018   ss: 0018
Process kfind (pid: 20927, stackpage=c52bd000)
Stack: c02414c0 c024159c 0000045c 0000003b c02415cc c52bc000 0000045c 00000000
       00000042 c012df1c c024159c 000001d2 00000006 000001d2 c0241780 000001ff 
       00000000 00000002 c012e750 000001d2 c52bc000 c0241780 c012ee6a 000001d2 
Call Trace: [<c012df1c>] [<c012e750>] [<c012ee6a>] [<c012f8d3>] [<c012f666>]
   [<c0126442>] [<c01264ba>] [<c0127b0b>] [<c0123b5d>] [<c0123e13>]
[<c01117ab>] 
   [<c0111694>] [<c01a7b0e>] [<c01a802c>] [<c01abb58>] [<c0109adc>]
[<c010bd08>] 
   [<c0109cc6>] [<c01088e4>]
Code: 0f 0b 4e 00 20 30 21 c0 8d b6 00 00 00 00 8b 46 18 a8 80 74

>>EIP; c012dd02 <page_launder_zone+59a/664>   <=====
Trace; c012df1c <page_launder+150/2c8>
Trace; c012e750 <do_try_to_free_pages+c/16c>
Trace; c012ee6a <try_to_free_pages+2a/38>
Trace; c012f8d3 <__alloc_pages+1a3/2b0>
Trace; c012f666 <_alloc_pages+16/18>
Trace; c0126442 <page_cache_read+8e/e0>
Trace; c01264ba <read_cluster_nonblocking+26/40>
Trace; c0127b0b <filemap_nopage+15b/250>
Trace; c0123b5d <do_no_page+71/250>
Trace; c0123e13 <handle_mm_fault+d7/178>
Trace; c01117ab <do_page_fault+117/430>
Trace; c0111694 <do_page_fault+0/430>
Trace; c01a7b0e <ide_do_request+29a/2e4>
Trace; c01a802c <ide_intr+16c/1ac>
Trace; c01abb58 <ide_dma_intr+0/a4>
Trace; c0109adc <handle_IRQ_event+34/60>
Trace; c010bd08 <end_8259A_irq+18/1c>
Trace; c0109cc6 <do_IRQ+ae/e4>
Trace; c01088e4 <error_code+34/40>
Code;  c012dd02 <page_launder_zone+59a/664>
00000000 <_EIP>:
Code;  c012dd02 <page_launder_zone+59a/664>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c012dd04 <page_launder_zone+59c/664>
   2:   4e                        dec    %esi
Code;  c012dd05 <page_launder_zone+59d/664>
   3:   00 20                     add    %ah,(%eax)
Code;  c012dd07 <page_launder_zone+59f/664>
   5:   30 21                     xor    %ah,(%ecx)
Code;  c012dd09 <page_launder_zone+5a1/664>
   7:   c0 8d b6 00 00 00 00      rorb   $0x0,0xb6(%ebp)
Code;  c012dd10 <page_launder_zone+5a8/664>
   e:   8b 46 18                  mov    0x18(%esi),%eax
Code;  c012dd13 <page_launder_zone+5ab/664>
  11:   a8 80                     test   $0x80,%al
Code;  c012dd15 <page_launder_zone+5ad/664>
  13:   74 00                     je     15 <_EIP+0x15> c012dd17
<page_launder_zone+5af/664>


4 warnings issued.  Results may not be reliable.



Was running KDE 3.0 attempting a kfind while updatedb was running in the
background.  System became unresponsive to mouse.  Attempting to switch to a vc
via alt-ctrl-f2 caused a redraw of the screen with kfind missing but no vc
switch.  At this point, life seemed back to normal.  Looking back at the logs
showed the following

Out of Memory: Killed process 21305 (httpd).

repeated many times...

Further back in the log was the following....

May 21 09:25:27 Camhanaich imapd[21311]: Command stream end of file, while
reading line user=??? host=turtle [192.168.0.2]
May 21 09:25:18 Camhanaich sendmail[1272]: rejecting connections on daemon
Daemon0: load average: 53
May 21 09:25:34 Camhanaich telnetd[21317]: ttloop: peer died: EOF 

On this sam box I also getthe following errors,in the past...

May 10 23:59:05 Camhanaich kernel: eth2: Error -110 writing Tx descriptor to
BAP
May 10 23:27:52 Camhanaich kernel: eth2: Error -110 setting multicast list.
May 10 23:27:53 Camhanaich kernel: eth2: Error -110 setting multicast list.
May 10 23:27:53 Camhanaich kernel: eth2: Error -110 setting PROMISCUOUSMODE to
1.
May 10 23:27:54 Camhanaich kernel: eth2: Error -110 writing Tx descriptor to
BAP

where eth2 is a linksys pc card in a pci adaptor using the orinoco_plx driver
(have tried the linux-wlan driver in the past with success, but it doesn't
compile currently with this kernel).  When the first mentioned problem
occurred, this card was disabled.

TIA
Erik

__________________________________________________
Do You Yahoo!?
LAUNCH - Your Yahoo! Music Experience
http://launch.yahoo.com

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[William Lee Irwin III]

On Tue, May 21, 2002 at 12:43:49PM -0700, Erik McKee wrote:
> Hello
> This output...
> kernel BUG at /usr/src/linux-2.2.13/include/linux/mm_inline.h:78!

Can I see the patch you used to merge preempt?


Cheers,
Bill

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[William Lee Irwin III]

On Tue, May 21, 2002 at 12:43:49PM -0700, Erik McKee wrote:
> Trace; c0123e13 <handle_mm_fault+d7/178>
> Trace; c01117ab <do_page_fault+117/430>
> Trace; c0111694 <do_page_fault+0/430>
> Trace; c01a7b0e <ide_do_request+29a/2e4>
> Trace; c01a802c <ide_intr+16c/1ac>
> Trace; c01abb58 <ide_dma_intr+0/a4>

I suspect a 2.4 IDE vs. preempt interaction, Andre, any chance you could
look at this?


Thanks,
Bill

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[Erik McKee]

It's preempt-kernel-rml-2.4.19-pre8-ac1-1.patch from
http://www.kernel.org/pub/linux/kernel/people/rml/preempt-kernel/linux-2.4/preempt-kernel-rml-2.4.19-pre8-ac1-1.patch

It applied cleanly with no mods needed and had been running fine untill this
decided to happen.  Seems like slocate's updatedb decided to jack the load up
which triggered oom?  However, the chosen process was unkillable since its the
same process listed in the oom report over and over again?


--- William Lee Irwin III <wli@holomorphy.com> wrote:
> On Tue, May 21, 2002 at 12:43:49PM -0700, Erik McKee wrote:
> > Hello
> > This output...
> > kernel BUG at /usr/src/linux-2.2.13/include/linux/mm_inline.h:78!
> 
> Can I see the patch you used to merge preempt?
> 
> 
> Cheers,
> Bill

__________________________________________________
Do You Yahoo!?
LAUNCH - Your Yahoo! Music Experience
http://launch.yahoo.com

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[William Lee Irwin III]

On Tue, May 21, 2002 at 01:23:51PM -0700, Erik McKee wrote:
> It's preempt-kernel-rml-2.4.19-pre8-ac1-1.patch from
> http://www.kernel.org/pub/linux/kernel/people/rml/preempt-kernel/linux-2.4/preempt-kernel-rml-2.4.19-pre8-ac1-1.patch
> It applied cleanly with no mods needed and had been running fine untill this
> decided to happen.  Seems like slocate's updatedb decided to jack the load up
> which triggered oom?  However, the chosen process was unkillable since its the
> same process listed in the oom report over and over again?

This looks to me like the IDE driver faulted in the interrupt handler,
everything after that is probably no more than just the usual system
state corruption from such events.


Cheers,
Bill

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[William Lee Irwin III]

On Tue, May 21, 2002 at 12:43:49PM -0700, Erik McKee wrote:
>> Trace; c0123e13 <handle_mm_fault+d7/178>
>> Trace; c01117ab <do_page_fault+117/430>
>> Trace; c0111694 <do_page_fault+0/430>
>> Trace; c01a7b0e <ide_do_request+29a/2e4>
>> Trace; c01a802c <ide_intr+16c/1ac>
>> Trace; c01abb58 <ide_dma_intr+0/a4>

On Tue, May 21, 2002 at 01:17:39PM -0700, William Lee Irwin III wrote:
> I suspect a 2.4 IDE vs. preempt interaction, Andre, any chance you could
> look at this?

Disregard that, brainfart. Continuing the hunt...


Cheers,
Bill

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[Alan Cox]

> It's preempt-kernel-rml-2.4.19-pre8-ac1-1.patch from
> http://www.kernel.org/pub/linux/kernel/people/rml/preempt-kernel/linux-2.4/preempt-kernel-rml-2.4.19-pre8-ac1-1.patch
> 
> It applied cleanly with no mods needed and had been running fine untill this
> decided to happen.  Seems like slocate's updatedb decided to jack the load up
> which triggered oom?  However, the chosen process was unkillable since its the
> same process listed in the oom report over and over again?

Can you repeat the problem without pre-empt ?

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[Robert Love]

On Tue, 2002-05-21 at 12:43, Erik McKee wrote:
> Hello
> 
> This output...
> <snip>

I don't think this has anything to do with preempt.  The current task
was not preemptible (hence the error notice on exit - is that why you
blame preempt?).  There is also no preempt_schedule call in your back
trace.

Looks to me like you died coming off an IDE interrupt and a resulting
read - you ran out of free pages and bit the dust there.  Dunno why,
though.  I don't have an mm_inline.h:78 in my tree, but I do have a
DEBUG_LRU near it ...

	Robert Love

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[Erik McKee]

And now for another data point.  Upon atempting to reboot after this, the raid5
array (software) was unable to be stopped.  Got as far as the lines....

Rebooting.
Stoping array md0

or something like that, and then hung....otherwise everything else worked fine
up between the bug and then.  Trying to reproduce without premept at the
moment.


--- Robert Love <rml@tech9.net> wrote:
> On Tue, 2002-05-21 at 12:43, Erik McKee wrote:
> > Hello
> > 
> > This output...
> > <snip>
> 
> I don't think this has anything to do with preempt.  The current task
> was not preemptible (hence the error notice on exit - is that why you
> blame preempt?).  There is also no preempt_schedule call in your back
> trace.
> 
> Looks to me like you died coming off an IDE interrupt and a resulting
> read - you ran out of free pages and bit the dust there.  Dunno why,
> though.  I don't have an mm_inline.h:78 in my tree, but I do have a
> DEBUG_LRU near it ...
> 
> 	Robert Love
> 


__________________________________________________
Do You Yahoo!?
LAUNCH - Your Yahoo! Music Experience
http://launch.yahoo.com

Re: Kernel BUG 2.4.19-pre8-ac1 + preempt

[Erik McKee]

Well here are the results of trying 2.4.19-pre8-ac5 with no preempt.  I have to
throw a lot of processes in kde at it to to get it to become unresponsive. 
Preformance becomes more sluggish as the number of processes and htus the load
is ramped up.  Eventially, I get down to 8k swap free, at which point system
becomes unresponsive.  Sendmail doesn't send, becuase of load avg near 40
something.  Clock on konsole doesn't move anywhere and the oom killer tries
again to kill stuff.  However, as far as i can see, no oops.  The first one was
only in dmesg and not log, so this one might have not made it through to syslog
(if it happened).  changing to vc is a no go at this point.  alt-sysrq-s seems
to work.  Tried alt-sysrq-k with no obvious effect.  At one point it seemed as
if i was really out of x and at a vc, however the x display stayed on he screen
and subsequent alt-sysrq-k's only caused the screen to move around, which could
be fixed by alt-ctrl-f2 to recenter the screen.  At this point the hd light had
finally gone out, perhaps as the result of previous alt-sysrq-k's, althouhg no
mention of them is found in the logs.  After a reboot and a resync of the raid
array, life seems bak to normal again...  Well, pretty much....  I did find
some corruption on the partition which is my boot partition.... a file missing,
e2fsck replayed the ext3 journal and foudn some errors, and tried copying the
file over, cp i/o error can't stat the file....  Alas, can't unmount the
partition, otherwise would reformat it, and reinstall lilo and my kernel images
on it.  SO off to try this via floppy rebooting ;)

JUst to dispell bad harddrive ?'s on this one, rformatting it definately worked
;)

--- Robert Love <rml@tech9.net> wrote:
> I don't think this has anything to do with preempt.  The current task
> was not preemptible (hence the error notice on exit - is that why you
> blame preempt?).  There is also no preempt_schedule call in your back
> trace.
> 
> Looks to me like you died coming off an IDE interrupt and a resulting
> read - you ran out of free pages and bit the dust there.  Dunno why,
> though.  I don't have an mm_inline.h:78 in my tree, but I do have a
> DEBUG_LRU near it ...
> 
> 	Robert Love
> 


__________________________________________________
Do You Yahoo!?
LAUNCH - Your Yahoo! Music Experience
http://launch.yahoo.com