list delete bug: kernel crash

list delete bug: kernel crash

[security]

Hi all,

I hope i mail in the right mailing list.


Sometime my kernel crash (i must reset my computer) and i have this logs:

Aug 30 20:42:24 gateway kernel: NF_IP_ASSERT:
net/ipv4/netfilter/ip_conntrack_core.c:1115(ip_conntrack_alter_reply)
Aug 30 20:42:24 gateway kernel: NF_IP_ASSERT:
net/ipv4/netfilter/ip_conntrack_core.c:1115(ip_conntrack_alter_reply)
Aug 30 20:44:24 gateway kernel: LIST_DELETE:
net/ipv4/netfilter/ip_conntrack_core.c:300
`&ct->tuplehash[IP_CT_DIR_REPLY]'(f3cedca4) not in
 &ip_conntrack_hash[hr].

I have search in bugzilla and google and, apparently, this bug has been solve
since kernel 2.6.5.
But i have kernel 2.6.8.1 and i have try kernel 2.6.6 and 2.6.7: same crash

But it hard to "see" because i can stay 1 week without crash, or 5mn.

I have the "local NAT" activated.

I have made a bugzilla report:
https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=236

If you need more informations, want me to made debug trace etc... i can, just
ask me and tell me how do that :)

Thanks.

Re: list delete bug: kernel crash

[KOVACS Krisztian]

  Hi,

2004-08-31, k keltezéssel 01:34-kor security ezt írta:
> Aug 30 20:42:24 gateway kernel: NF_IP_ASSERT:
> net/ipv4/netfilter/ip_conntrack_core.c:1115(ip_conntrack_alter_reply)
> Aug 30 20:42:24 gateway kernel: NF_IP_ASSERT:
> net/ipv4/netfilter/ip_conntrack_core.c:1115(ip_conntrack_alter_reply)
> Aug 30 20:44:24 gateway kernel: LIST_DELETE:
> net/ipv4/netfilter/ip_conntrack_core.c:300
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(f3cedca4) not in
>  &ip_conntrack_hash[hr].
> 
> I have search in bugzilla and google and, apparently, this bug has been solve
> since kernel 2.6.5.
> But i have kernel 2.6.8.1 and i have try kernel 2.6.6 and 2.6.7: same crash
> 
> But it hard to "see" because i can stay 1 week without crash, or 5mn.
> 
> I have the "local NAT" activated.

  As always, the first thing you should try is testing the memory of
your computer. The problems seems to be caused by trying to call
ip_nat_setup_info() on an already confirmed connection, which is known
to cause hash corruption.

  BTW, while running memtest, could you send us the nat table of your
iptables ruleset, along with the routing setup? And also a list of
loaded (iptables-related) kernel modules would be useful.

-- 
 Regards,
   Krisztian KOVACS

Re: list delete bug: kernel crash

[security]

>
>  As always, the first thing you should try is testing the memory of
> your computer. The problems seems to be caused by trying to call
> ip_nat_setup_info() on an already confirmed connection, which is known
> to cause hash corruption.
>

I have launch memtest86 and, surprise, i have got memory error. I think i 
have found the faulty module memory (no more memtest86 error when i leave it 
from my computer).
So i will test some day to see if i got no more crash.

I didn't think about memory because all crash was on netfilter and never with 
something else.

thanks for all. 

Re: list delete bug: kernel crash

[security]

----- Original Message ----- 
From: "security" <security@lea-linux.com>
To: "KOVACS Krisztian" <hidden@balabit.hu>
Cc: <netfilter@lists.netfilter.org>
Sent: Tuesday, August 31, 2004 1:39 PM
Subject: Re: list delete bug: kernel crash


>
>>
>>  As always, the first thing you should try is testing the memory of
>> your computer. The problems seems to be caused by trying to call
>> ip_nat_setup_info() on an already confirmed connection, which is known
>> to cause hash corruption.
>>
>
> I have launch memtest86 and, surprise, i have got memory error. I think i 
> have found the faulty module memory (no more memtest86 error when i leave 
> it from my computer).
> So i will test some day to see if i got no more crash.
>

Hum still have crash. I have test again my memory, made 10 pass with 
memtest86 without error found.

>>BTW, while running memtest, could you send us the nat table of your
>>iptables ruleset, along with the routing setup? And also a list of
>>loaded (iptables-related) kernel modules would be useful.


Here are informations:

-----------------------------
NAT:
-----------------------------

/usr/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o ppp0 -j 
MASQUERADE

#Bittorent redirect to 192.168.0.10

iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 6881:6889 -j 
DNAT --to-dest 192.168.0.10
iptables -A FORWARD -p tcp -i ppp0 --dport 6881:6889 -d 192.168.0.10 -j 
ACCEPT
iptables -t nat -A PREROUTING -i ppp0 -p udp --dport 6881:6889 -j 
DNAT --to-dest 192.168.0.10
iptables -A FORWARD -p udp -i ppp0 --dport 6881:6889 -d 192.168.0.10 -j 
ACCEPT

# nat module for ftp and irc

modprobe ip_nat_ftp
modprobe ip_nat_irc





----------------------------------
Firewall rules
----------------------------------

# Flush all rules in chains
iptables -F

#delete all user's chains
iptables -X

#Forward rules

#Create a new chain: KEEP_STATE
iptables -N KEEP_STATE
iptables -F KEEP_STATE

#Drop packet in bad states
iptables -A KEEP_STATE -m state --state INVALID -j DROP

#Accept packet in good states
iptables -A KEEP_STATE -m state --state RELATED,ESTABLISHED -j ACCEPT

#deny bad packet and log them
iptables -A FORWARD -p tcp --tcp-flags ALL FIN,URG,PSH -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "NMAP-XMAS: "
iptables -A FORWARD -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
iptables -A FORWARD -p tcp --tcp-flags SYN,FIN SYN,FIN -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "SYN/FIN: "
iptables -A FORWARD -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN,RST -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "SYN/RST: "
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN,RST -j DROP

#Drop RST/ACKs to limit OS detection throught pinging
iptables -A FORWARD -p tcp --tcp-flags RST RST,ACK -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "RST/ACK: "
iptables -A FORWARD -p tcp --tcp-flags RST RST,ACK -j DROP


#drop possible directory traversal port
iptables -A FORWARD -p tcp --dport 2301 -j DROP

#deny pings from outside and accept local / network
iptables -A FORWARD -p icmp --icmp-type 0/0 -d 192.168.0.0/24 -j ACCEPT
iptables -A FORWARD -p icmp --icmp-type 0/0 -m limit --limit 5/minute -j 
LOG --log-level notice --log-prefix "Drop Echo Reply: "
iptables -A FORWARD -p icmp --icmp-type 0/0 -j DROP

#reject identd to avoid timeout on irc connect
iptables -A FORWARD -p tcp --dport 113 -j REJECT
#Pass all boxes to the keep_state chain
iptables -A FORWARD -j KEEP_STATE


#################### BLOCK SPECIFIC HOSTS #######################
iptables -A FORWARD -s 194.237.107.150 -j DROP
iptables -A FORWARD -d 194.237.107.150 -j DROP


#Allow outgoing traffic
iptables -A FORWARD -s 192.168.0.0/24 -j ACCEPT

#Deny all traffic not define by any rules
iptables -A FORWARD -j DROP

## own gateway input/ouput rules

#deny bad packet and log them
iptables -A INPUT -p tcp --tcp-flags ALL FIN,URG,PSH -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "NMAP-XMAS: "
iptables -A INPUT -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "SYN/FIN: "
iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -m limit --limit 
5/minute -j LOG --log-level notice --log-prefix "SYN/RST: "
iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP

#Drop RST/ACKs to limit OS detection throught pinging
iptables -A INPUT -p tcp --tcp-flags RST RST,ACK -m limit --limit 5/minute -j 
LOG --log-level notice --log-prefix "RST/ACK: "
iptables -A INPUT -p tcp --tcp-flags RST RST,ACK -j DROP


#drop possible directory traversal port
iptables -A INPUT -p tcp --dport 2301 -j DROP
iptables -A OUTPUT -p icmp --icmp-type 0/0 -d 192.168.0.0/24 -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type 0/0 -d 127.0.0.0/24 -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type 0/0 -m limit --limit 5/minute -j 
LOG --log-level notice --log-prefix "Drop icmp echo:"
iptables -A OUTPUT -p icmp --icmp-type 0/0 -j DROP


#################### BLOCK SPECIFIC HOSTS #######################
iptables -A INPUT -s 194.237.107.150 -j DROP
iptables -A INPUT -d 194.237.107.150 -j DROP
iptables -A OUTPUT -s 194.237.107.150 -j DROP
iptables -A OUTPUT -d 194.237.107.150 -j DROP


## irc behavior
iptables -A INPUT -p tcp --dport 113 -j REJECT

#Pass all in keep_state
iptables -A INPUT -j KEEP_STATE

#Allow SSH input/output
iptables -A INPUT -p tcp  --dport 22 -j ACCEPT
iptables -A INPUT -p tcp  --sport 22 -j ACCEPT
#Allow all local traffic
iptables -A INPUT -p tcp -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -p udp -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -p icmp -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -p tcp -s 192.168.0.0/24 -j ACCEPT
iptables -A INPUT -p udp -s 192.168.0.0/24 -j ACCEPT
iptables -A INPUT -p icmp -s 192.168.0.0/24 -j ACCEPT

#Allow ping and traceroute from this host, reply for staff adsl
iptables -A INPUT -p icmp --icmp-type 0/0 -i ppp0 -j ACCEPT

#Allow DC
#iptables -A INPUT -p tcp --sport 14567 -j ACCEPT
#iptables -A INPUT -p udp --sport 14567 -j ACCEPT
iptables -A INPUT -p tcp --dport 14567 -j ACCEPT
iptables -A INPUT -p udp --dport 14567 -j ACCEPT
#Allow DNS from this host

#Block netbios
iptables -A INPUT -i ppp0 -p tcp --dport 139 -j REJECT
iptables -A INPUT -p tcp --sport 53 -j ACCEPT
iptables -A INPUT -p udp --sport 53 -j ACCEPT

#Allow webmin & web
iptables -A INPUT -p tcp --dport 10000 -j ACCEPT
#iptables -A INPUT -p tcp --dport 80 -j ACCEPT

#Allow smtp
iptables -A INPUT -p tcp --dport 25 -j ACCEPT
#Allow cvs
iptables -A INPUT -p tcp --dport 2401 -j ACCEPT
#Allow stream server
iptables -A INPUT -p tcp --dport 8090 -j ACCEPT
#Allow ftp from this host
iptables -A INPUT -p tcp --sport 21 -j ACCEPT
iptables -A INPUT -p tcp --sport 20 -j ACCEPT

#Allow dhcpd for local network
iptables -A INPUT -i eth0 -p tcp --sport 68 --dport 67 -j ACCEPT
iptables -A INPUT -i eth0 -p udp --sport 68 --dport 67 -j ACCEPT

#Allow emule for natting
iptables -A INPUT -p tcp -s 0/0 --dport 4661 -j ACCEPT
iptables -A INPUT -p tcp -s 0/0 --dport 4662 -j ACCEPT
iptables -A INPUT -p tcp -s 0/0 --dport 4711 -j ACCEPT
iptables -A INPUT -p udp -s 0/0 --dport 4665 -j ACCEPT
iptables -A INPUT -p udp -s 0/0 --dport 4672 -j ACCEPT

#Deny all other
iptables -A INPUT -j DROP

Re: list delete bug: kernel crash

[Gavin Hamill]

On Tuesday 31 August 2004 14:17, security wrote:

> Hum still have crash. I have test again my memory, made 10 pass with
> memtest86 without error found.

This may be a long shot, but there may exist the possibility of the files on 
disk being corrupted when they were installed, due to your faulty memory?

Try to reinstall the kernel / modules and iptables userspace?

Cheers,
Gavin.

Re: list delete bug: kernel crash

[security]

>
> This may be a long shot, but there may exist the possibility of the files 
> on
> disk being corrupted when they were installed, due to your faulty memory?
>
> Try to reinstall the kernel / modules and iptables userspace?
>
> Cheers,
> Gavin.
>
>
Ok. I go re-compil my kernel and re-install iptable package :) 

Re: list delete bug: kernel crash

[security]

----- Original Message ----- 
From: "security" <security@lea-linux.com>
To: "Gavin Hamill" <gdh@acentral.co.uk>; <netfilter@lists.netfilter.org>
Sent: Tuesday, August 31, 2004 3:34 PM
Subject: Re: list delete bug: kernel crash


> >
>> This may be a long shot, but there may exist the possibility of the files 
>> on
>> disk being corrupted when they were installed, due to your faulty memory?
>>
>> Try to reinstall the kernel / modules and iptables userspace?
>>
>> Cheers,
>> Gavin.
>>
>>
> Ok. I go re-compil my kernel and re-install iptable package :)

Ok still have  crash after re-compil my kernel and re-install iptable 
userspace package.
Allways the same error:

Aug 31 16:01:39 gateway kernel: LIST_DELETE: 
net/ipv4/netfilter/ip_conntrack_core.c:300 
`&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9224) not in
&ip_conntrack_hash[hr].
Aug 31 16:01:39 gateway kernel: LIST_DELETE: 
net/ipv4/netfilter/ip_conntrack_core.c:300 
`&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9524) not in
&ip_conntrack_hash[hr].
Aug 31 16:03:26 gateway kernel: LIST_DELETE: 
net/ipv4/netfilter/ip_conntrack_core.c:300 
`&ct->tuplehash[IP_CT_DIR_REPLY]'(d6537224) not in
&ip_conntrack_hash[hr].

Re: list delete bug: kernel crash

[Gavin Hamill]

On Tuesday 31 August 2004 15:18, security wrote:

> Ok still have  crash after re-compil my kernel and re-install iptable
> userspace package.
> Allways the same error:
>
> Aug 31 16:01:39 gateway kernel: LIST_DELETE:
> net/ipv4/netfilter/ip_conntrack_core.c:300
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9224) not in
> &ip_conntrack_hash[hr].

That's a shame. Unless your netfilter support came via modules rather than 
from the kernel itself, then I'm afraid I have no more suggestions :( My last 
thinking was that you may not have replaced any modules - only the main 
kernel file...

Cheers,
Gavin.

Re: list delete bug: kernel crash

[security]

----- Original Message ----- 
From: "Gavin Hamill" <gdh@acentral.co.uk>
To: <netfilter@lists.netfilter.org>
Sent: Tuesday, August 31, 2004 5:05 PM
Subject: Re: list delete bug: kernel crash


> On Tuesday 31 August 2004 15:18, security wrote:
>
>> Ok still have  crash after re-compil my kernel and re-install iptable
>> userspace package.
>> Allways the same error:
>>
>> Aug 31 16:01:39 gateway kernel: LIST_DELETE:
>> net/ipv4/netfilter/ip_conntrack_core.c:300
>> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9224) not in
>> &ip_conntrack_hash[hr].
>
> That's a shame. Unless your netfilter support came via modules rather than
> from the kernel itself, then I'm afraid I have no more suggestions :( My 
> last
> thinking was that you may not have replaced any modules - only the main
> kernel file...
>

I have re-compil / install kernel and modules.
But, i will wait because when the last crash occur, i had re install iptable 
userspace BUT didn't have reboot.
And i have notice too that the iptable userspace i used was 1.2.11 compiled 
by me, and now i have re-install offciel slackware 10.0 package: 1.2.10
So i will wait to see if i have other crash since i have reboot with this 
iptable userspace version. 

Re: list delete bug: kernel crash

[Alistair Tonner]

On August 31, 2004 10:18 am, security wrote:
> ----- Original Message -----
> From: "security" <security@lea-linux.com>
> To: "Gavin Hamill" <gdh@acentral.co.uk>; <netfilter@lists.netfilter.org>
> Sent: Tuesday, August 31, 2004 3:34 PM
> Subject: Re: list delete bug: kernel crash
>
> >> This may be a long shot, but there may exist the possibility of the
> >> files on
> >> disk being corrupted when they were installed, due to your faulty
> >> memory?
> >>
> >> Try to reinstall the kernel / modules and iptables userspace?
> >>
> >> Cheers,
> >> Gavin.
> >
> > Ok. I go re-compil my kernel and re-install iptable package :)
>
> Ok still have  crash after re-compil my kernel and re-install iptable
> userspace package.
> Allways the same error:
>
> Aug 31 16:01:39 gateway kernel: LIST_DELETE:
> net/ipv4/netfilter/ip_conntrack_core.c:300
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9224) not in
> &ip_conntrack_hash[hr].
> Aug 31 16:01:39 gateway kernel: LIST_DELETE:
> net/ipv4/netfilter/ip_conntrack_core.c:300
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9524) not in
> &ip_conntrack_hash[hr].
> Aug 31 16:03:26 gateway kernel: LIST_DELETE:
> net/ipv4/netfilter/ip_conntrack_core.c:300
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d6537224) not in
> &ip_conntrack_hash[hr].

	I recall having memory problems in the past (about three years ago) ... 
	never very fun, and the issue was so fine grained that I ended up having to 
rebuild the box from a zeroed disk.  Reading backward I see you are using 
2.6.8.1 kernel code.  Can you check 

	1) the MD5 sum of the tarball of kernel code and the 
	2)MD5 sum of the tarball of iptables, 
	
	just as a quick verification that they are (close to) clean.  

	next == which version of iptables ( i didn't notice that in your original 
post) and what  elements if any out of patch-o-matic(-ng) are installed?

	Keep in mind that you now have to question any code that is on your system 
that might have been built wilst that damaged memory module was installed, 
one never knows where a bit might have been flipped. *sigh*

	This particular message is from LIST_DELETE in function
	clean_from_lists() and appears at first glance to be the cleanup after 
expectation timeout.  You don't have any tweaks to any of the (expectation) 
timeout code somewhere do you?
	
	You could *try* running MD5sum against ip_*.c 
in /usr/src/linux/net/ipv4/netfilter dir -- and *possibly* someone could 
verify the numbers  .....but I  personally would redownload the whole lot to 
be safe  (i.e. kernel code/iptables code etc) -- if this is a recently built 
box, I'd rebuild from the ground up based on the bad memory module... but I'm 
paranoid....

	Alistair Tonner

	

	

Re: list delete bug: kernel crash

[Jose Maria Lopez]

El mar, 31 de 08 de 2004 a las 16:18, security escribió:
> ----- Original Message ----- 
> From: "security" <security@lea-linux.com>
> To: "Gavin Hamill" <gdh@acentral.co.uk>; <netfilter@lists.netfilter.org>
> Sent: Tuesday, August 31, 2004 3:34 PM
> Subject: Re: list delete bug: kernel crash
> 
> 
> > >
> >> This may be a long shot, but there may exist the possibility of the files 
> >> on
> >> disk being corrupted when they were installed, due to your faulty memory?
> >>
> >> Try to reinstall the kernel / modules and iptables userspace?
> >>
> >> Cheers,
> >> Gavin.
> >>
> >>
> > Ok. I go re-compil my kernel and re-install iptable package :)
> 
> Ok still have  crash after re-compil my kernel and re-install iptable 
> userspace package.
> Allways the same error:
> 
> Aug 31 16:01:39 gateway kernel: LIST_DELETE: 
> net/ipv4/netfilter/ip_conntrack_core.c:300 
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9224) not in
> &ip_conntrack_hash[hr].
> Aug 31 16:01:39 gateway kernel: LIST_DELETE: 
> net/ipv4/netfilter/ip_conntrack_core.c:300 
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d3ac9524) not in
> &ip_conntrack_hash[hr].
> Aug 31 16:03:26 gateway kernel: LIST_DELETE: 
> net/ipv4/netfilter/ip_conntrack_core.c:300 
> `&ct->tuplehash[IP_CT_DIR_REPLY]'(d6537224) not in
> &ip_conntrack_hash[hr].

Have you tried to patch with patch-o-matic and recompile both
the kernel and the userspace last sources (iptables). Both sides
of the system must be synced or there will be problems.

-- 
Jose Maria Lopez Hernandez
Director Tecnico de bgSEC
jkerouac@bgsec.com
bgSEC Seguridad y Consultoria de Sistemas Informaticos
http://www.bgsec.com
ESPAÑA

The only people for me are the mad ones -- the ones who are mad to live,
mad to talk, mad to be saved, desirous of everything at the same time,
the ones who never yawn or say a commonplace thing, but burn, burn, burn
like fabulous yellow Roman candles.
                -- Jack Kerouac, "On the Road"

Re: list delete bug: kernel crash

[security]

>
> I recall having memory problems in the past (about three years ago) ...
> never very fun, and the issue was so fine grained that I ended up having to
> rebuild the box from a zeroed disk.  Reading backward I see you are using
> 2.6.8.1 kernel code.  Can you check
>

[snip]

I have backup all my data, take my 2 install slackware CDROM, format all my 
disk and re-install slackware 10.0.
Now i'm sure:
- My memory is not faulty (10 PASS of memtest86 without error)
- kernel / program are installed / compiled without memory fault
- i have a fresh and good install

No crash since my re install ,in fact i'm still installing, so many things to 
do... and as usuall i have forget to backup some files, but it's ok lol :)

So if i got another kernel crash i will post here :)

thanks all to help me :)