kernel freeze issue

kernel freeze issue

[Zoltan Nagy]

summary of current configuration:

           +----- ext_if(eth1) 3c905TX ------ 34Mbit uplink
           |
           |
           |
         +------------+
         |   filter   +
         +------------+
           |  |  |
           |  |  +--- lan_if(eth5) rtl8169 - vlan1 ------ users/servers
           |  |
           |  +------ adm_if(eth0) 3c905TX - vlan2 ------ log server
           |
           +--------- core_if(eth3) 3c940
                      +inp_if(eth3.3)      - vlan 3  +
                      +out_if(eth3.4)      - vlan 4  +
                                                     |
                                                     |(cross-link cable)
                                                     |
               +------ core_if(eth0) 3c940 ----------+
               |
         +-----------+
         |    foo    +
         +-----------+
               |
               +------ adm_if(eth1) rtl8139 - vlan2


filter:
	arp proxy based
	route		-	policy routing - between local-domain1(C),local-domain2(2*C),uplink,foo(on failover this skipped)
	packet filter(netfilter)
				-	traffic accounting(ipt_account),flood/portscan protection
				-	packet filter
				-	TTL inc
				-	ipset's for extra port configurations
				-	ipt_condition(failover control)
				-	we have 2 domain's so it sends redirects for the hosts spoofing that it's our router(ipt_IPALTER)
foo:
	not configured because of the freezes...

problem:
	filter is freezes in random intervalls(30m - 6day) - on-board watchdog(i8xx) reboot's the system
	i've tryed many things, remove my custom patches...but it won't help ;)
	in the kernel trace i've last seen(i've a blurry image)
	the kernel remove some packages from the boomerang interface
	ip_rcv_finish, etc..
	ipt_do_table is the last in the call trace...

notice:
	crash happens when many of our beloved users use p2p software(this is also a tip)

next try:
	place a cisco to monitor ext_if and lan_if with tcpdump, open another file every 10m
	and when filter freezes i maybe have the packet that caused the freeze
	(small chance - but possible ;)

my tips was:
	ipt_condition	-	in pom it's <2.6.0 but i've read the code, and i think it's safe to use
	ipt_IPALTER	-	w/o it also freezes, so this isn't the problem
	ipset		-	i've a small patch on it...to enable inverted portmaps, i think it's safe
					i use portmap,ipmap,macipmap from it
	NAPI			-	yesterday i disabled it...since then no freezes
	ipt_TCPMSS	-	it wrote some warnings in dmesg, about packet size<64 - i've moved another rule before it
						 -p tcp --tcp-option ! 2    -j DROP
	boomerang		-	maybe the driver is a bit broken
	pom_patches	-	TTL set connmark CONNMARK account condition limit
	SMP			-	maybe, i haven't disabled it yet

today's surprise:
	ip l s eth0 promisc on	==> freeze, without any trace ;)

versions:
Linux filter 2.6.14-alt #5 SMP Tue Nov 8 16:40:49 CET 2005 i686 GNU/Linux
iptables-1.3.4
pom-20051031
ipset-2.2.6

some info about the system can be downloaded from
http://152.66.235.5/info-filter.tgz
this is my worst nightmare, any suggestions? ;)

Re: kernel freeze issue

[KOVACS Krisztian]

  Hi,

On Wednesday 02 November 2005 21.45, Zoltan Nagy wrote:
> problem:
> 	filter is freezes in random intervalls(30m - 6day) - on-board
> watchdog(i8xx) reboot's the system i've tryed many things, remove my
> custom patches...but it won't help ;) in the kernel trace i've last
> seen(i've a blurry image)
> 	the kernel remove some packages from the boomerang interface
> 	ip_rcv_finish, etc..
> 	ipt_do_table is the last in the call trace...

  Please try to get at least a backtrace, withouth that probably nobody will 
be able to help you. The above information (ip_rcv_finish and ipt_do_table 
being present in the trace) is completely useless, as processing _any_ IPv4 
packet could lead to receiving such a backtrace...

> my tips was:
> 	ipt_condition	-	in pom it's <2.6.0 but i've read the code, and i think
> it's safe to use 

> today's surprise:
> 	ip l s eth0 promisc on	==> freeze, without any trace ;)

  You could still try getting a backtrace with SysRq + P.

-- 
 Regards,
  Krisztian Kovacs

Re: kernel freeze issue

[/dev/rob0]

General comment on the format: it would help if you would put the text 
and list portions in standard (>80 columns) format. ASCII drawings are 
harder to fit sometimes, but even those will come across better under 
80 columns. I use a GUI MUA so I was able to resize to view your post 
properly, but remember, much of your audience might be using mutt.

On Wednesday 2005-November-02 14:45, Zoltan Nagy wrote:
> problem:
> 	filter is freezes in random intervalls(30m - 6day) - on-board
> watchdog(i8xx) reboot's the system i've tryed many things, remove my
> custom patches...but it won't help ;) in the kernel trace i've last

I would suspect hardware first.

> crash happens when many of our beloved users use p2p software(this
> is also a tip)
snip
> today's surprise:
> 	ip l s eth0 promisc on	==> freeze, without any trace ;)

Did you try swapping out that NIC?

> some info about the system can be downloaded from
> http://152.66.235.5/info-filter.tgz
> this is my worst nightmare, any suggestions? ;)

You do have my sympathy, but I didn't download the file. I would be 
likely to do that if it was something I could view in the browser, like 
text or HTML.
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Re: kernel freeze issue

[Zoltan Nagy]

KOVACS Krisztian wrote:
>>today's surprise:
>>	ip l s eth0 promisc on	==> freeze, without any trace ;)
> 
> 
>   You could still try getting a backtrace with SysRq + P.
> 

tried, sysrq not responding...

regards

Re: kernel freeze issue

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

On Wed, Nov 09, 2005 at 05:06:44PM +0100, Zoltan Nagy wrote:
> 
> 
> KOVACS Krisztian wrote:
> >>today's surprise:
> >>	ip l s eth0 promisc on	==> freeze, without any trace ;)
> > 
> > 
> >   You could still try getting a backtrace with SysRq + P.
> > 
> 
> tried, sysrq not responding...
> 
> regards

try booting with these kernel parameters
nmi_watchdog=1 console=ttyS0,9600n8 console=/dev/tty0

it will setup nmi_watchdog and will also send all you syslog stuff out
to the serial port.

Then connect serial port to another machine to capture the oops/crash

> 
> 
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: kernel freeze issue

[Zoltan Nagy]

last night, netconsole captured a trace ;)
i'm not an expert in interpreting kernel traces
any suggestions?

regard.kirk

Nov 10 19:11:45 filter netconsole: network logging started
Nov 10 23:41:47 filter Unable to handle kernel paging request
Nov 10 23:41:47 filter  at virtual address e0fd5000
Nov 10 23:41:47 filter  printing eip:
Nov 10 23:41:47 filter c03fb036
Nov 10 23:41:47 filter *pde = 1fe27067
Nov 10 23:41:47 filter *pte = 00000000
Nov 10 23:41:47 filter Oops: 0000 [#1]
Nov 10 23:41:47 filter SMP
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter Modules linked in:
Nov 10 23:41:47 filter  netconsole
Nov 10 23:41:47 filter  i8xx_tco
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter CPU:    0
Nov 10 23:41:47 filter EIP:    0060:[<c03fb036>]    Not tainted VLI
Nov 10 23:41:47 filter EFLAGS: 00010213   (2.6.14-alt)
Nov 10 23:41:47 filter EIP is at match+0x586/0x8b0
Nov 10 23:41:47 filter eax: 00000001   ebx: 00000800   ecx: 00000800
edx: 00000000
Nov 10 23:41:47 filter esi: e0fd3000   edi: e0fd3000   ebp: e0fbb000
esp: c04f7d8c
Nov 10 23:41:47 filter ds: 007b   es: 007b   ss: 0068
Nov 10 23:41:47 filter Process swapper (pid: 0, threadinfo=c04f6000
task=c048cbe0)
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter Stack:
Nov 10 23:41:47 filter 979b87c9
Nov 10 23:41:47 filter 00000800
Nov 10 23:41:47 filter 00000048
Nov 10 23:41:47 filter e10268c6
Nov 10 23:41:47 filter c012a46d
Nov 10 23:41:47 filter c04b44a0
Nov 10 23:41:47 filter 00000282
Nov 10 23:41:47 filter 00000000
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter 00000020
Nov 10 23:41:47 filter 000002ee
Nov 10 23:41:47 filter e0fc8500
Nov 10 23:41:47 filter 0000001e
Nov 10 23:41:47 filter 085975c0
Nov 10 23:41:47 filter e0fbb0d8
Nov 10 23:41:47 filter e0fbd000
Nov 10 23:41:47 filter e0fd6000
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter 000007ff
Nov 10 23:41:47 filter 000007fe
Nov 10 23:41:47 filter 000007fe
Nov 10 23:41:47 filter e0fd3000
Nov 10 23:41:47 filter 6c025a8c
Nov 10 23:41:47 filter 6bf3acd9
Nov 10 23:41:47 filter 003d26b1
Nov 10 23:41:47 filter 00000000
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter Call Trace:
Nov 10 23:41:47 filter  [<c012a46d>]
Nov 10 23:41:47 filter  notifier_call_chain+0x2d/0x50
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c03f1cd8>]
Nov 10 23:41:47 filter  ipt_do_table+0x1d8/0x370
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c03b3260>]
Nov 10 23:41:47 filter  ip_rcv_finish+0x0/0x270
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c03f7ef6>]
Nov 10 23:41:47 filter  ipt_hook+0x36/0x40
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c040cfc8>]
Nov 10 23:41:47 filter  nf_iterate+0x78/0x90
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c03b3260>]
Nov 10 23:41:47 filter  ip_rcv_finish+0x0/0x270
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c03b3260>]
Nov 10 23:41:47 filter  ip_rcv_finish+0x0/0x270
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c040d04e>]
Nov 10 23:41:47 filter  nf_hook_slow+0x6e/0x110
Nov 10 23:41:47 filter
Nov 10 23:41:47 filter  [<c03b3260>]
Nov 10 23:41:47 filter  ip_rcv_finish+0x0/0x270
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c03b3260>]
Nov 10 23:41:48 filter  ip_rcv_finish+0x0/0x270
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c03b2fcb>]
Nov 10 23:41:48 filter  ip_rcv+0x40b/0x530
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c03b3260>]
Nov 10 23:41:48 filter  ip_rcv_finish+0x0/0x270
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c039cd78>]
Nov 10 23:41:48 filter  netif_receive_skb+0x1c8/0x270
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c039ceb2>]
Nov 10 23:41:48 filter  process_backlog+0x92/0x120
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c039cfd5>]
Nov 10 23:41:48 filter  net_rx_action+0x95/0x190
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c0121c16>]
Nov 10 23:41:48 filter  __do_softirq+0xd6/0xf0
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c0121c65>]
Nov 10 23:41:48 filter  do_softirq+0x35/0x40
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c010522e>]
Nov 10 23:41:48 filter  do_IRQ+0x1e/0x30
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c01039e6>]
Nov 10 23:41:48 filter  common_interrupt+0x1a/0x20
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c0100db0>]
Nov 10 23:41:48 filter  default_idle+0x0/0x30
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c0100dd3>]
Nov 10 23:41:48 filter  default_idle+0x23/0x30
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c0100e90>]
Nov 10 23:41:48 filter  cpu_idle+0x70/0x80
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c04f899f>]
Nov 10 23:41:48 filter  start_kernel+0x15f/0x180
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter  [<c04f8390>]
Nov 10 23:41:48 filter  unknown_bootoption+0x0/0x1e0
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter Code:
Nov 10 23:41:48 filter 84
Nov 10 23:41:48 filter 80
Nov 10 23:41:48 filter 01
Nov 10 23:41:48 filter 00
Nov 10 23:41:48 filter 00
Nov 10 23:41:48 filter 8b
Nov 10 23:41:48 filter 5c
Nov 10 23:41:48 filter 24
Nov 10 23:41:48 filter 48
Nov 10 23:41:48 filter 8b
Nov 10 23:41:48 filter 0d
Nov 10 23:41:48 filter 4c
Nov 10 23:41:48 filter f5
Nov 10 23:41:48 filter 55
Nov 10 23:41:48 filter c0
Nov 10 23:41:48 filter 8b
Nov 10 23:41:48 filter 74
Nov 10 23:41:48 filter 24
Nov 10 23:41:48 filter 4c
Nov 10 23:41:48 filter 43
Nov 10 23:41:48 filter 89
Nov 10 23:41:48 filter d8
Nov 10 23:41:48 filter 99
Nov 10 23:41:48 filter f7
Nov 10 23:41:48 filter f9
Nov 10 23:41:48 filter 83
Nov 10 23:41:48 filter 3c
Nov 10 23:41:48 filter 96
Nov 10 23:41:48 filter ff
Nov 10 23:41:48 filter 0f
Nov 10 23:41:48 filter 84
Nov 10 23:41:48 filter be
Nov 10 23:41:48 filter 00
Nov 10 23:41:48 filter last message repeated 2 times
Nov 10 23:41:48 filter 89
Nov 10 23:41:48 filter 4c
Nov 10 23:41:48 filter 24
Nov 10 23:41:48 filter 04
Nov 10 23:41:48 filter 8b
Nov 10 23:41:48 filter 7c
Nov 10 23:41:48 filter 24
Nov 10 23:41:48 filter 4c
Nov 10 23:41:48 filter b>
Nov 10 23:41:48 filter 14
Nov 10 23:41:48 filter 9f
Nov 10 23:41:48 filter 8d
Nov 10 23:41:48 filter 04
Nov 10 23:41:48 filter d5
Nov 10 23:41:48 filter 00
Nov 10 23:41:48 filter last message repeated 3 times
Nov 10 23:41:48 filter 29
Nov 10 23:41:48 filter d0
Nov 10 23:41:48 filter 8b
Nov 10 23:41:48 filter 54
Nov 10 23:41:48 filter 24
Nov 10 23:41:48 filter 38
Nov 10 23:41:48 filter 8b
Nov 10 23:41:48 filter 04
Nov 10 23:41:48 filter 82
Nov 10 23:41:48 filter 89
Nov 10 23:41:48 filter 04
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter
Nov 10 23:41:48 filter Kernel panic - not syncing: Fatal exception in
interrupt
Nov 10 23:41:48 filter

Re: kernel freeze issue

[Zoltan Nagy]

hi

/dev/rob0 wrote:
> General comment on the format: it would help if you would put the text 
> and list portions in standard (>80 columns) format. ASCII drawings are 
> harder to fit sometimes, but even those will come across better under 
> 80 columns. I use a GUI MUA so I was able to resize to view your post 
> properly, but remember, much of your audience might be using mutt.
> 

ok, i use 212x83 terminals ;)

> On Wednesday 2005-November-02 14:45, Zoltan Nagy wrote:
> 
>>problem:
>>	filter is freezes in random intervalls(30m - 6day) - on-board
>>watchdog(i8xx) reboot's the system i've tryed many things, remove my
>>custom patches...but it won't help ;) in the kernel trace i've last
> 
> 
> I would suspect hardware first.
another part of the story...problem started with a rtl8139 if
on the side of the micro it have been failed time to time,
the transciever has gone mad, and packet traversal has become 20% ;)
so...we tried a 3c940 to put it on the router side - the card don't
take up the link...
we tried rtl8169 - like the 940
so, we bought 2 brand new 3c905tx, i'm not sure(because at this time
was a hw change, and i've rewrote nearly everything in a new style...
so i don't suspected hw issue, but i will try an intel100 ;)


> 
> 
> Did you try swapping out that NIC?
> 
> 
yes ;)

> You do have my sympathy, but I didn't download the file. I would be 
> likely to do that if it was something I could view in the browser, like 
> text or HTML.
i think people like to read source in there own ways, i prefer use less..
/there's a bit too many files in it/
...and i hate viewing sources in browser - i hate white bg color ;)

regards.kirk