Willy Tarreau wrote:
> On Wed, May 24, 2006 at 09:32:20AM -0400, Steve Clark wrote:
> 
>>Hello,
>>
>>I realize this is uClinux - but I received no response from the 
>>uclinux-devel list so I am
>>posting here in the hope that someone will be kind enough to at least 
>>give me a
>>direction in which to proceed in trying to track down this panic. It 
>>only seems to
>>happen under heavy traffic - especially outbound, in other words 
>>ethernet to modem to internet.
>>
>>I am having trouble with cnxtserial.c causing a panic on an actiontec 
>>dualpc
>>modem. I've looked at the code but can't figure out how I can be 
>>getting a null pointer. I have very
>>limited experience working in the kernel so if anyone can give 
>>somethings to try I would appreciate
>>it. Below is the output from ksymoops of the panic.
> 
> 
> Could you please inform us about :
>   - the architecture (not obvious from the dump)
>   - the *exact* kernel version. uclinux and 2.4.32 are not enough. I
>     would guess 2.4.32-uc0, but I'm not sure that it's the only one.
>     Also, if you have other patches applied, please point to them.
>   - a list of the loaded modules and the .config
>   - how often it happens (every minutes, hours, days, weeks, ...) and
>     if you know an easy way to reproduce it (you only mentionned that
>     it *only* seems to happen in the circumstances above, but not if
>     it's guaranteed).
> 
> Of course, this will not guarantee that anybody will reply nor find
> the bug, but right now, it might discourage many people from searching.
> 
> 
>>Thanks,
>>Steve
> 
> 
> Regards,
> Willy
> 
> 
> 
>>ksymoops 2.4.11 on i686 2.6.16-1.2122_FC5.  Options used
>>      -v /home/sclark/actiontec/linux-2.4.32/vmlinux (specified)
>>      -K (specified)
>>      -L (specified)
>>      -O (specified)
>>      -m ../System.map (specified)
>>      -t elf32-littlearm -a armnommu
>>
>># Unable to handle kernel NULL pointer dereference at virtual address 
>>0000003f
>>Internal error: Oops: ffffffff
>>CPU: 0
>>pc : [<0090e5fc>]    lr : [<008727cc>]    Not tainted
>>sp : 0098bc14  ip : 7e17685d  fp : 0098bc38
>>r10: 00a10000  r9 : 173706d5  r8 : 800e000a
>>r7 : 08010100  r6 : 00dc08d0  r5 : 24108090  r4 : c914d625
>>r3 : 6c9c0a91  r2 : 00000018  r1 : 009922b8  r0 : c3400038
>>Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  Segment user
>>Control: C000107D
>>Process ksoftirqd_CPU0 (pid: 3, stackpage=0098b000)
>>0098bc00:          008727cc 0090e5fc 20000093  ffffffff 00000038 
>>00000000 00992
>>0098bc20: 00000038 00956228 20000013 0098bc6c  0098bc3c 008727cc 
>>0090e3b0 00000
>>0098bc40: 00000000 00992200 00000038 0099221c  0094ce68 00000000 
>>00000001 00a10
>>0098bc60: 0098bc98 0098bc70 008820e0 00872694  00992200 00a29760 
>>00a29760 00994
>>0098bc80: 00000034 00000000 009ea9e2 0098bcb0  0098bc9c 00882004 
>>00882024 00994
>>0098bca0: 00000000 0098bcc8 0098bcb4 008794f4  00881fd4 0000002f 
>>00a29760 0098b
>>0098bcc0: 0098bccc 008793c0 00879434 0098bcd4  00000001 009ea9e2 
>>00994d64 00994
>>0098bce0: 00000000 0094ce68 00994d60 00000000  0098bd1c 0098bd00 
>>00878e04 00878
>>0098bd00: 00a24080 009edec0 009edec0 00992400  0098bd44 0098bd20 
>>00878b60 00878
>>0098bd20: 00a24080 00992400 009edec0 00000000  008b7744 00000002 
>>0098bd60 0098b
>>0098bd40: 008a9c74 0087891c 0094ce68 0094ce60  00992400 0098bd80 
>>0098bd64 0089e
>>0098bd60: 008a9c1c 00000001 009edec0 00000000  00000000 0098bd9c 
>>0098bd84 008b7
>>0098bd80: 0089ea64 00000001 00000004 00992400  0098bde0 0098bda0 
>>008a8fec 008b7
>>0098bda0: 00992400 0098bdb0 008b7744 80000000  00964ac0 009edec0 
>>009edec0 009ed
>>0098bdc0: 00992400 0093ded4 008b4204 00000000  00000002 0098be08 
>>0098bde4 008b5
>>0098bde0: 008a8ea4 00992400 008b7744 80000000  009edec0 00000002 
>>00992400 0098b
>>0098be00: 0098be0c 008b4258 008b5730 00000001  0098be60 0098be20 
>>008a8fec 008b4
>>0098be20: 00992400 0098be30 008b4204 80000000  00964ab0 009edec0 
>>009edef0 009ed
>>0098be40: 000005f4 00992400 008b29e0 00000000  00000002 0098be8c 
>>0098be64 008b4
>>0098be60: 008a8ea4 00992400 008b4204 80000000  00a243a0 0093ded4 
>>009edec0 0093d
>>0098be80: 0098bebc 0098be90 008b2c08 008b3f38  0093ded4 00000005 
>>00000000 00000
>>0098bea0: 0093ded4 00000001 00000000 00000000  0098bf00 0098bec0 
>>008a8fec 008b2
>>0098bec0: 00000000 0098bed0 008b29e0 80000000  00964aa0 009edec0 
>>00000005 009ed
>>0098bee0: 009edec0 00965500 0093ded4 00036356  009381a0 0098bf38 
>>0098bf04 008b2
>>0098bf00: 008a8ea4 00000000 008b29e0 80000000  009edec0 0093fc1c 
>>009edec0 00000
>>0098bf20: 00000008 00000001 0098bf88 0098bf54  0098bf3c 0089f54c 
>>008b2284 0093d
>>0098bf40: 009381cc 009381ac 0098bf84 0098bf58  0089f65c 0089f2ec 
>>00000040 00938
>>0098bf60: 009381cc 009381a0 009381bc 00036356  60000013 0094d224 
>>0098bfb0 0098b
>>0098bf80: 0089f7dc 0089f5c4 0000012c 00938090  00000003 fffffff2 
>>0094ce60 00000
>>0098bfa0: 00938080 0098bfdc 0098bfb4 0081e090  0089f754 0098a000 
>>0098a000 0098a
>>0098bfc0: 0094ce60 00938d48 41029402 0080ae58  0098bffc 0098bfe0 
>>0081e70c 0081d
>>0098bfe0: 00000000 0097e000 0081e674 00945730  00000000 0098c000 
>>00810f28 0081e
>>Backtrace:
>>Function entered at [<0090e3a0>] from [<008727cc>]
>>  r9 = 20000013  r8 = 00956228  r7 = 00000038  r6 = 009922A0
>>  r5 = 00000000  r4 = 00000038
>>Function entered at [<00872684>] from [<008820e0>]
>>Function entered at [<00882014>] from [<00882004>]
>>Function entered at [<00881fc4>] from [<008794f4>]
>>  r5 = 00000000  r4 = 00994D60
>>Function entered at [<00879424>] from [<008793c0>]
>>  r5 = 00A29760  r4 = 0000002F
>>Function entered at [<00878ebc>] from [<00878e04>]
>>Function entered at [<00878d60>] from [<00878b60>]
>>  r7 = 00992400  r6 = 009EDEC0  r5 = 009EDEC0  r4 = 00A24080
>>Function entered at [<0087890c>] from [<008a9c74>]
>>Function entered at [<008a9c0c>] from [<0089ebf0>]
>>  r6 = 00992400  r5 = 0094CE60  r4 = 0094CE68
>>Function entered at [<0089ea54>] from [<008b781c>]
>>  r7 = 00000000  r6 = 00000000  r5 = 009EDEC0  r4 = 00000001
>>Function entered at [<008b7744>] from [<008a8fec>]
>>  r6 = 00992400  r5 = 00000004  r4 = 00000001
>>Function entered at [<008a8e94>] from [<008b58cc>]
>>Function entered at [<008b5720>] from [<008b4258>]
>>  r6 = 00992400  r5 = 00000002  r4 = 009EDEC0
>>Function entered at [<008b4204>] from [<008a8fec>]
>>  r4 = 00000001
>>Function entered at [<008a8e94>] from [<008b413c>]
>>Function entered at [<008b3f28>] from [<008b2c08>]
>>  r7 = 0093DED4  r6 = 009EDEC0  r5 = 0093DED4  r4 = 00A243A0
>>Function entered at [<008b29e0>] from [<008a8fec>]
>>  r6 = 00000000  r5 = 00000000  r4 = 00000001
>>Function entered at [<008a8e94>] from [<008b2768>]
>>Function entered at [<008b2274>] from [<0089f54c>]
>>  r8 = 0098BF88  r7 = 00000001  r6 = 00000008  r5 = 00000000
>>  r4 = 009EDEC0
>>Function entered at [<0089f2dc>] from [<0089f65c>]
>>  r6 = 009381AC  r5 = 009381CC  r4 = 0093DED4
>>Function entered at [<0089f5b4>] from [<0089f7dc>]
>>Function entered at [<0089f744>] from [<0081e090>]
>>Function entered at [<0081dfd8>] from [<0081e70c>]
>>Function entered at [<0081e674>] from [<00810f28>]
>>  r7 = 00945730  r6 = 0081E674  r5 = 0097E000  r4 = 00000000
>>Code: ba000003 e93113f8 (e92013f8) e2522020 aafffffb
>>Error (Oops_bfd_perror): /tmp/ksymoops.BYmaYB Invalid bfd target
>>
>>
>> >>LR;  008727cc <rs_write+148/294>
>>
>> >>PC;  0090e5fc <memmove+25c/460>   <=====
>>
>>Trace; 0090e3a0 <memcpy+0/0>
>>Trace; 008727cc <rs_write+148/294>
>>
>> >>r8; 00956228 <tmp_buf+0/1000>
>>
>>Trace; 00872684 <rs_write+0/294>
>>Trace; 008820e0 <ppp_async_push+cc/230>
>>Trace; 00882014 <ppp_async_push+0/230>
>>Trace; 00882004 <ppp_async_send+40/50>
>>Trace; 00881fc4 <ppp_async_send+0/50>
>>Trace; 008794f4 <ppp_push+d0/1b8>
>>Trace; 00879424 <ppp_push+0/1b8>
>>Trace; 008793c0 <ppp_send_frame+504/568>
>>Trace; 00878ebc <ppp_send_frame+0/568>
>>Trace; 00878e04 <ppp_xmit_process+a4/15c>
>>Trace; 00878d60 <ppp_xmit_process+0/15c>
>>Trace; 00878b60 <ppp_start_xmit+254/2c8>
>>Trace; 0087890c <ppp_start_xmit+0/2c8>
>>Trace; 008a9c74 <.gcc2_compiled.+68/100>
>>Trace; 008a9c0c <qdisc_restart+0/0>
>>Trace; 0089ebf0 <dev_queue_xmit+19c/3b0>
>>
>> >>r5; 0094ce60 <irq_stat+0/20>
>> >>r4; 0094ce68 <irq_stat+8/20>
>>
>>Trace; 0089ea54 <dev_queue_xmit+0/3b0>
>>Trace; 008b781c <ip_finish_output2+d8/150>
>>Trace; 008b7744 <ip_finish_output2+0/150>
>>Trace; 008a8fec <nf_hook_slow+158/1f0>
>>Trace; 008a8e94 <nf_hook_slow+0/1f0>
>>Trace; 008b58cc <ip_finish_output+1ac/1b8>
>>Trace; 008b5720 <ip_finish_output+0/1b8>
>>Trace; 008b4258 <ip_forward_finish+54/98>
>>Trace; 008b4204 <ip_forward_finish+0/98>
>>Trace; 008a8fec <nf_hook_slow+158/1f0>
>>Trace; 008a8e94 <nf_hook_slow+0/1f0>
>>Trace; 008b413c <.gcc2_compiled.+214/2dc>
>>Trace; 008b3f28 <ip_forward+0/0>
>>Trace; 008b2c08 <ip_rcv_finish+228/29c>
>>
>> >>r7; 0093ded4 <eth0_dev+0/164>
>> >>r5; 0093ded4 <eth0_dev+0/164>
>>
>>Trace; 008b29e0 <ip_rcv_finish+0/29c>
>>Trace; 008a8fec <nf_hook_slow+158/1f0>
>>Trace; 008a8e94 <nf_hook_slow+0/1f0>
>>Trace; 008b2768 <ip_rcv+4f4/568>
>>Trace; 008b2274 <ip_rcv+0/568>
>>Trace; 0089f54c <netif_receive_skb+270/2d8>
>>Trace; 0089f2dc <netif_receive_skb+0/2d8>
>>Trace; 0089f65c <process_backlog+a8/190>
>>
>> >>r6; 009381ac <softnet_data+c/1a0>
>> >>r5; 009381cc <softnet_data+2c/1a0>
>> >>r4; 0093ded4 <eth0_dev+0/164>
>>
>>Trace; 0089f5b4 <process_backlog+0/190>
>>Trace; 0089f7dc <net_rx_action+98/1bc>
>>Trace; 0089f744 <net_rx_action+0/1bc>
>>Trace; 0081e090 <__do_softirq+b8/15c>
>>Trace; 0081dfd8 <.gcc2_compiled.+0/0>
>>Trace; 0081e70c <ksoftirqd+98/cc>
>>Trace; 0081e674 <ksoftirqd+0/cc>
>>Trace; 00810f28 <arch_kernel_thread+38/44>
>>
>> >>r7; 00945730 <__machine_arch_type+0/4>
>> >>r6; 0081e674 <ksoftirqd+0/cc>
>>
>>Kernel panic: Aiee, killing interrupt handler
>>
>>1 error issued.  Results may not be reliable.
>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
> 
> 
> 
Thanks for the tip Willy,

The hardware is the ActionTec DualPC Modem it has a conexant cx82100 
arm processor.
I can reproduce it at will by connecting to the internet and running 
nttcp thru it at the same time
I am scping file both ways from and to, and then finally starting a 
getty on /dev/ttyS0, the modem is
at ttyS1 and also ttyS0 is where all the kernel printk messages come out.

When I start the getty if I have all the other traffic going it 
usually will panic in under a minute. IF I don't
have the getty running it will run for hours and not panic.



2.4.32-uc0 with patches from
http://www.bettina-attack.de/jonny/view.php/projects/uclinux_on_cx82100/