More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jonathan Cooper]

Specifically, the message:

kernel: LIST_DELETE: ip_conntrack_core.c:302
&ct->tuplehash[IP_CT_DIR_REPLY]'(d40acea4) not in ip_conntrack_hash[hr].

From doing a bit of googling it appears this problem has been around for
some time; hopefully my situation will help you track it down.

The machine exhibiting the problem has 2 network cards: an nForce2 card
(eth1) on the motherboard (connected to external network) and a sis900
(eth0) which is connected via crossover cable to another computer. This
connection is masqueraded.

It would appear that trying to access the machine via the masqueraded
interface leads to the message being produced after a delay of perhaps 2
minutes. Certainly this is the case when trying to access a webserver or
webcache on the machine directly via that interface. The connection times
out and shortly afterwards the message appears in syslog. DHCP & ICMP
appear to work OK. Also interestingly you will note from the iptables -L
output shown below that I have squid set up as a transparent proxy for
the masqueraded network using a redirect rule in the prerouting chain -
the computer that is being masqueraded can access webpages from the
outside world, so the redirect seems to avoid the problem.

Under the 2.4.23 and 2.4.24 kernels my machine would oops on reboot (just
after killing syslog) after the message had occurred. This has not (yet)
happened on 2.4.25. One common feature, that has occurred once under
2.4.23 and once under 2.4.25 is that after several occurrences of this
message the computer froze solid, with no keyboard, mouse or network
response.

I have experienced no problems of this nature with Debian's 2.4.22
kernel.

Experiments I have performed so far include:

* Delete the MASQUERADE rule. Same problem, although have not yet tried
this fresh after a reboot.

* Add an ACCEPT rule to the top of the FORWARD chain. Same problem. Have
tried this fresh after a reboot.

* Flush the FORWARD chain. Same problem, although have not yet tried
this fresh after a reboot.


(Under 2.4.25 I did notice that one failed connection appeared to be
producing multiple LIST_DELETE messages. However I did have the http
connection in question going through a local squid, so it could be that
that was making multiple tries to fetch the page. Stopping proxying for
local addresses appears to have returned us to the previously noted
behaviour.)


Excepts from my iptables configuration:
# iptables -L -n
[snip]
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
LOG        all  --  0.0.0.0/0            0.0.0.0/0           state
  INVALID LOG flags 0 level 4 prefix 'invalid-state: '
DROP       all  --  0.0.0.0/0            0.0.0.0/0           state
  INVALID
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state
  RELATED,ESTABLISHED
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:6112
good-bad   all  --  192.168.1.0/24       0.0.0.0/0
good-vpn   all  --  192.168.1.0/24       0.0.0.0/0
vpn-good   all  --  0.0.0.0/0            0.0.0.0/0
bad-good   all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  192.168.168.0/24     0.0.0.0/0
LOG        all  --  0.0.0.0/0            0.0.0.0/0           LOG flags 0
  level 4 prefix 'chain-jump: '
DROP       all  --  0.0.0.0/0            0.0.0.0/0
[snip]

# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
REDIRECT   tcp  --  192.168.1.0/24       0.0.0.0/0           tcp dpt:80
  redir ports 8080
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:6112
  to:192.168.1.2:6112

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  192.168.1.0/24       0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

(The DNAT rule is to allow the other computer to create StarCraft games
on Battle.net.)


Output from scripts/ver_linux:
Linux finarfin 2.4.25 #1 Wed Mar 10 12:32:06 GMT 2004 i686 GNU/Linux

Gnu C                  3.3.3
Gnu make               3.80
binutils               2.14.90.0.7
util-linux             2.12
mount                  2.12
modutils               2.4.26
e2fsprogs              1.35-WIP
PPP                    2.4.2
Linux C Library        2.3.2
Dynamic linker (ldd)   2.3.2
Procps                 3.2.0
Net-tools              1.60
Console-tools          0.2.3
Sh-utils               5.0.91
Modules Loaded         nvidia parport_pc lp parport apm snd-seq-oss
snd-seq-midi-event snd-seq snd-pcm-oss snd-mixer-oss snd-intel8x0 snd-pcm
snd-timer snd-ac97-codec snd-page-alloc snd-mpu401-uart snd-rawmidi
snd-seq-device snd soundcore ipt_limit ipt_REJECT ipt_MASQUERADE
ipt_REDIRECT ipt_LOG ipt_state iptable_nat ip_conntrack iptable_filter
ip_tables nls_cp437 vfat fat nls_iso8859-1 ntfs forcedeth ide-scsi
scsi_mod ide-cd cdrom keybdev input usb-ohci usbcore rtc


My 2.4.25 kernel config is at <http://jonc.me.uk/stuff/config-2.4.25>

Let me know if there's any other information you want, or any tests that
you would like me to run.

Jonathan.

-- 
Jonathan Cooper        MSN: msn@jonc.me.uk        jonc.me.uk/

Spotted at the railway station:
  Passengers are asked not to cross the lines - it takes ages for us to
  uncross them again.

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jonathan Cooper]

* Jonathan Cooper <netfilter@jonc.me.uk> [2004-03-11 16:18]:
> Under the 2.4.23 and 2.4.24 kernels my machine would oops on reboot (just
> after killing syslog) after the message had occurred. This has not (yet)
> happened on 2.4.25. One common feature, that has occurred once under
> 2.4.23 and once under 2.4.25 is that after several occurrences of this
> message the computer froze solid, with no keyboard, mouse or network
> response.

Update: only one occurrence of the message is required to trigger a
freeze, it seems, although the freeze occurred some time after the
message. Looks suspiciously like random memory corruption...

> I have experienced no problems of this nature with Debian's 2.4.22
> kernel.

The config of which is now available at
<http://jonc.me.uk/stuff/config-2.4.22>.

Jonathan.

-- 
Jonathan Cooper        MSN: msn@jonc.me.uk        jonc.me.uk/

Health: The slowest possible rate of dying.

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Kovacs Krisztian]

  Hi,

On Thu, Mar 11, 2004 at 04:50:37PM +0000, Jonathan Cooper wrote:
> The config of which is now available at
> <http://jonc.me.uk/stuff/config-2.4.22>.

  Could you try turning CONFIG_IP_NF_NAT_LOCAL on?

-- 
 KOVACS Krisztian

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jonathan Cooper]

* Kovacs Krisztian <piglet@sch.bme.hu> [2004-03-12 00:04]:
>   Could you try turning CONFIG_IP_NF_NAT_LOCAL on?

I've just tried this, and find I can now fetch pages via the masqueraded
interface from the machine itself, with no occurrences of LIST_DELETE
messages. Fetching pages from my other computer on the masqueraded
network also works.

So this seems to fix the problem. I'll keep the kernel running with debug
on for a bit just in case.

Jonathan.

-- 
Jonathan Cooper        MSN: msn@jonc.me.uk        jonc.me.uk/

A clear concience is usually the sign of a bad memory.

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[KOVACS Krisztian]

  Hi,

On Thu, 2004-03-11 at 22:17, Kovacs Krisztian wrote:
> On Thu, Mar 11, 2004 at 04:50:37PM +0000, Jonathan Cooper wrote:
> > The config of which is now available at
> > <http://jonc.me.uk/stuff/config-2.4.22>.
> 
>   Could you try turning CONFIG_IP_NF_NAT_LOCAL on?

  Jozsef, and others pursuing this ugly thing. Please take a look at
http://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014442.html
and probably the thread about find_appropriate_src(). I'd be really glad
to have someone answer it, or somehow show some interest in it.

-- 
 Regards,
   Krisztian KOVACS

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jozsef Kadlecsik]

On Fri, 12 Mar 2004, KOVACS Krisztian wrote:

>   Jozsef, and others pursuing this ugly thing. Please take a look at
> http://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014442.html
> and probably the thread about find_appropriate_src(). I'd be really glad
> to have someone answer it, or somehow show some interest in it.

I'll try to look at it. Unfortunately at the moment I'm very busy with
issues from the institute I work for. Sigh. Hopefully I'll find some
time-slot...

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[KOVACS Krisztian]

  Hi,

On Fri, 2004-03-12 at 14:30, Jozsef Kadlecsik wrote:
> >   Jozsef, and others pursuing this ugly thing. Please take a look at
> > http://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014442.html
> > and probably the thread about find_appropriate_src(). I'd be really glad
> > to have someone answer it, or somehow show some interest in it.
> 
> I'll try to look at it. Unfortunately at the moment I'm very busy with
> issues from the institute I work for. Sigh. Hopefully I'll find some
> time-slot...

  I can reliably reproduce the bug. One of the IPs of the host is
10.1.0.1. Only one NAT rule is needed:

# iptables -t nat -A PREROUTING -p tcp -s 10.1.0.0/16 --dport 80 -j
REDIRECT --to-ports 8080

  Then try to connect to port 80 of 10.1.0.1. The connection gets stuck
in SYN_SENT state, and when it times out, corrupts its REPLY_DIR hash
chain. Note that LOCAL_NAT is _off_, so this connection won't have a
null binding allocated.

-- 
 Regards,
   Krisztian KOVACS

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Vijaya Chandra Vupputuri]

Hi,

've tried to do the same but I don't seem to have any problem on a test
box with redhat 7.3 (2.4.18-3)

If I understood what you said properly the following is your setup,
the ip of the test box is 10.1.0.1 and you have a lan 10.1.0.0/16 whose
gateway is 10.1.0.1
on 10.1.0.1 you redirect any traffic to port 80 to the local port 8080

now if 10.1.0.2 tries to connect to, say, 216.239.41.104 it gets
redirected to the port 8080 on 10.1.0.1

but, from 10.1.0.2, if you try to connect directly to port 80 on
10.1.0.1, with your setup where nothing is listening on 80, the box
would hang. am i right?!

please do let me know if i've missed anything obvious.

Regards,
Vijaya Chandra Vupputuri,
Tachyon Technologies.

On Tue, 2004-03-16 at 22:58, KOVACS Krisztian wrote:
>   Hi,
> 
> On Fri, 2004-03-12 at 14:30, Jozsef Kadlecsik wrote:
> > >   Jozsef, and others pursuing this ugly thing. Please take a look at
> > > http://lists.netfilter.org/pipermail/netfilter-devel/2004-March/014442.html
> > > and probably the thread about find_appropriate_src(). I'd be really glad
> > > to have someone answer it, or somehow show some interest in it.
> > 
> > I'll try to look at it. Unfortunately at the moment I'm very busy with
> > issues from the institute I work for. Sigh. Hopefully I'll find some
> > time-slot...
> 
>   I can reliably reproduce the bug. One of the IPs of the host is
> 10.1.0.1. Only one NAT rule is needed:
> 
> # iptables -t nat -A PREROUTING -p tcp -s 10.1.0.0/16 --dport 80 -j
> REDIRECT --to-ports 8080
> 
>   Then try to connect to port 80 of 10.1.0.1. The connection gets stuck
> in SYN_SENT state, and when it times out, corrupts its REPLY_DIR hash
> chain. Note that LOCAL_NAT is _off_, so this connection won't have a
> null binding allocated.

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Henrik Nordstrom]

On Wed, 17 Mar 2004, Vijaya Chandra Vupputuri wrote:

> 've tried to do the same but I don't seem to have any problem on a test
> box with redhat 7.3 (2.4.18-3)

And you have built your kernel with CONFIG_IP_NF_LOCAL_NAT disabled in
your kernel config? This is required to enable the assymetry in the NAT
code mentioned by KOVACS.

At least in the RedHat 2.4.20-28.7 kernel CONFIG_IP_NF_LOCAL_NAT is
enabled by default, so unless you have built a custom RedHat kernel with
this disabled you should not see the issue mentioned.

Regards
Henrik

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[KOVACS Krisztian]

  Hi,

On Wed, 2004-03-17 at 10:38, Vijaya Chandra Vupputuri wrote:
> 've tried to do the same but I don't seem to have any problem on a test
> box with redhat 7.3 (2.4.18-3)

  I think the problem is present only on kernels > 2.4.22. This is
because of a fix in 2.4.23 which changed when alloc_null_binding() is
called. In 2.4.22, alloc_null_binding() is called even when
CONFIG_IP_NF_LOCAL_NAT is off, while in 2.4.23 (and above) it's only
called when CONFIG_IP_NF_LOCAL_NAT is turned on. I don't know if it
matters or not with respect to this problem.

> If I understood what you said properly the following is your setup,
> the ip of the test box is 10.1.0.1 and you have a lan 10.1.0.0/16 whose
> gateway is 10.1.0.1
> on 10.1.0.1 you redirect any traffic to port 80 to the local port 8080

  Yes, this is the setup. (Although this is a virtual setup using UML,
but it shouldn't matter at all.)

> now if 10.1.0.2 tries to connect to, say, 216.239.41.104 it gets
> redirected to the port 8080 on 10.1.0.1

  Yes, in theory. To trigger the bug, you should connect _from_ the
gateway to itself. (Note that the ruleset is somewhat flawed, since it
should redirect only traffic coming in through the LAN interface.) I
used netcat to connect:

# nc 10.1.0.1 80

> but, from 10.1.0.2, if you try to connect directly to port 80 on
> 10.1.0.1, with your setup where nothing is listening on 80, the box
> would hang. am i right?!

  It does not matter if there is something listening on port 80. I think
it wasn't clear from my previous mail that you have to connect _from_
the gateway, and not from the attached 10.1.0.0/16 LAN.

-- 
 Regards,
   Krisztian KOVACS

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[KOVACS Krisztian]

  Hi,

On Wed, 2004-03-17 at 12:16, KOVACS Krisztian wrote:
>   Yes, in theory. To trigger the bug, you should connect _from_ the
> gateway to itself. (Note that the ruleset is somewhat flawed, since it
> should redirect only traffic coming in through the LAN interface.) I
> used netcat to connect:
> 
> # nc 10.1.0.1 80

  I think I've got it. When such a connection is initiated, the SYN goes
out on the LOCAL_OUT hook, and gets confirmed at POSTROUTING. Then it
arrives in PREROUTING, where it's NAT-ted according to the REDIRECT
rule. And here is the problem: ip_conntrack_alter_reply() modifies the
reply tuple of an already confirmed (= hashed) connection. This is the
place where we corrupt the hash.

  When CONFIG_IP_NF_LOCAL_NAT is on, alloc_null_binding() is called on
LOCAL_OUT, so that ip_nat_fn() will skip the packet when it is seen on
PREROUTING.

-- 
 Regards,
   Krisztian KOVACS

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Henrik Nordstrom]

On Wed, 17 Mar 2004, KOVACS Krisztian wrote:

>   I think I've got it. When such a connection is initiated, the SYN goes
> out on the LOCAL_OUT hook, and gets confirmed at POSTROUTING. Then it
> arrives in PREROUTING, where it's NAT-ted according to the REDIRECT
> rule.

Ouch... this (new NAT mappings of an already confirmed session) MUST NOT
happen.

A quick and painless fix unless one wants to enable CONFIG_IP_NF_LOCAL_NAT
would be to not apply NAT to confirmed sessions even if no NAT mapping has
priorly been set up for this connection.

But I am still a little confused... this traffic should have been seen by
the POSTROUTING hook where a null source mapping should have been set up
already for this direction, so the PREROUTING on return traffic should do
nothing. Thinking.. ah, now I get it. The problem is that you are doing
traffic local->local and that the NAT code then gets confused as it sees
the same packet twice in each direction, and as no null mapping was
created the first time for the "local" side PREROUTING tries to when it
sees the traffic coming "in" (from itself).

The solution I mentioned above should workI think, but my gut feeling is
that it would be better to simply have the hooks of CONFIG_IP_NF_LOCAL_NAT
always enabled and not try to optimize local traffic by making it such
special case for NAT to deal with.

Regards
Henrik

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[KOVACS Krisztian]

  Hi,

On Wed, 2004-03-17 at 14:04, Henrik Nordstrom wrote:
> nothing. Thinking.. ah, now I get it. The problem is that you are doing
> traffic local->local and that the NAT code then gets confused as it sees
> the same packet twice in each direction, and as no null mapping was
> created the first time for the "local" side PREROUTING tries to when it
> sees the traffic coming "in" (from itself).

  Yes, this is the case.

> The solution I mentioned above should workI think, but my gut feeling is
> that it would be better to simply have the hooks of CONFIG_IP_NF_LOCAL_NAT
> always enabled and not try to optimize local traffic by making it such
> special case for NAT to deal with.

  I agree. BTW, this is the patch which removed the nat hook on
LOCAL_OUT:

http://lists.netfilter.org/pipermail/netfilter-devel/2003-October/012617.html

-- 
 Regards,
   Krisztian KOVACS

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Henrik Nordstrom]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 613 bytes --]

On Wed, 17 Mar 2004, KOVACS Krisztian wrote:

> On Wed, 2004-03-17 at 14:04, Henrik Nordstrom wrote:
> > nothing. Thinking.. ah, now I get it. The problem is that you are doing
> > traffic local->local and that the NAT code then gets confused as it sees
> > the same packet twice in each direction, and as no null mapping was
> > created the first time for the "local" side PREROUTING tries to when it
> > sees the traffic coming "in" (from itself).
> 
>   Yes, this is the case.

The attached patch MAY fix this issue (completely untested, I always 
compile with CONFIG_IP_NF_LOCAL_NAT enabled).

Regards
Henrik

[-- Attachment #2: Type: TEXT/PLAIN, Size: 728 bytes --]

--- ip_nat_standalone.c-2.4.25	Wed Mar 17 17:23:33 2004
+++ ip_nat_standalone.c	Wed Mar 17 17:33:28 2004
@@ -114,7 +114,16 @@
 		WRITE_LOCK(&ip_nat_lock);
 		/* Seen it before?  This can happen for loopback, retrans,
 		   or local packets.. */
-		if (!(info->initialized & (1 << maniptype))) {
+		if (!(info->initialized & (1 << maniptype))
+#ifndef CONFIG_IP_NF_NAT_LOCAL
+		    /* If this session has already been confirmed we must not
+		     * touch it again even if there is no mapping set up.
+		     * Can only happen on local->local traffic with
+		     * CONFIG_IP_NF_NAT_LOCAL disabled.
+		     */
+		    && ct->status & IPS_CONFIRMED
+#endif
+		    ) {
 			unsigned int ret;
 
 			if (ct->master

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[KOVACS Krisztian]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]


  Hi,

On Wed, 2004-03-17 at 17:35, Henrik Nordstrom wrote:
> > On Wed, 2004-03-17 at 14:04, Henrik Nordstrom wrote:
> > > nothing. Thinking.. ah, now I get it. The problem is that you are doing
> > > traffic local->local and that the NAT code then gets confused as it sees
> > > the same packet twice in each direction, and as no null mapping was
> > > created the first time for the "local" side PREROUTING tries to when it
> > > sees the traffic coming "in" (from itself).
> > 
> >   Yes, this is the case.
> 
> The attached patch MAY fix this issue (completely untested, I always 
> compile with CONFIG_IP_NF_LOCAL_NAT enabled).

  I think the patch is buggy. The check should be

  if (!(info->initialized & (1 << maniptype))
      && !(ct->status & IPS_CONFIRMED))

instead of

  if (!(info->initialized & (1 << maniptype))
      && ct->status & IPS_CONFIRMED)

  Fixed patch attached. I've tested it on my UML testbed with stock
2.4.24, and seems to solve the problem.

-- 
 Regards,
   Krisztian KOVACS

[-- Attachment #2: renat_confirmed.patch --]
[-- Type: text/plain, Size: 709 bytes --]

--- ip_nat_standalone.c.orig	Fri Nov 28 19:26:21 2003
+++ ip_nat_standalone.c	Thu Mar 25 16:43:29 2004
@@ -114,7 +114,16 @@
 		WRITE_LOCK(&ip_nat_lock);
 		/* Seen it before?  This can happen for loopback, retrans,
 		   or local packets.. */
-		if (!(info->initialized & (1 << maniptype))) {
+		if (!(info->initialized & (1 << maniptype))
+#ifndef CONFIG_IP_NF_NAT_LOCAL
+		    /* If this session has already been confirmed we must not
+		     * touch it again even if there is no mapping set up.
+		     * Can only happen on local->local traffic with
+		     * CONFIG_IP_NF_NAT_LOCAL disabled.
+		     */
+		    && !(ct->status & IPS_CONFIRMED)
+#endif
+		    ) {
 			unsigned int ret;
 
 			if (ct->master

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Henrik Nordstrom]

On Thu, 25 Mar 2004, KOVACS Krisztian wrote:

>   I think the patch is buggy. The check should be
> 
>   if (!(info->initialized & (1 << maniptype))
>       && !(ct->status & IPS_CONFIRMED))
> 
> instead of
> 
>   if (!(info->initialized & (1 << maniptype))
>       && ct->status & IPS_CONFIRMED)


Indeed. I must have been a bit sleepy when I wrote that first patch..

Regards
Henrik

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Vijaya Chandra Vupputuri]

Hi,
I was lazy enough to send a mail to the list instead of rebooting my
test box with a 2.4.23 kernel ;) ('ve had actually disabled conntrack
for the 2.4.23 kernel, so i'd done the test previously only with the
default redhat kernel)

This time with 2.4.23, after trying to make the local connection, I
tried viewing the conntrack entries (/proc/net/ip_conntrack) and the
system happily crashed.

Regards,
Vijaya Chandra Vupputuri,
Tachyon Technologies.

On Wed, 2004-03-17 at 16:46, KOVACS Krisztian wrote:
>   Hi,
> 
> On Wed, 2004-03-17 at 10:38, Vijaya Chandra Vupputuri wrote:
> > 've tried to do the same but I don't seem to have any problem on a test
> > box with redhat 7.3 (2.4.18-3)
> 
>   I think the problem is present only on kernels > 2.4.22. This is
> because of a fix in 2.4.23 which changed when alloc_null_binding() is
> called. In 2.4.22, alloc_null_binding() is called even when
> CONFIG_IP_NF_LOCAL_NAT is off, while in 2.4.23 (and above) it's only
> called when CONFIG_IP_NF_LOCAL_NAT is turned on. I don't know if it
> matters or not with respect to this problem.
> 
> > If I understood what you said properly the following is your setup,
> > the ip of the test box is 10.1.0.1 and you have a lan 10.1.0.0/16 whose
> > gateway is 10.1.0.1
> > on 10.1.0.1 you redirect any traffic to port 80 to the local port 8080
> 
>   Yes, this is the setup. (Although this is a virtual setup using UML,
> but it shouldn't matter at all.)
> 
> > now if 10.1.0.2 tries to connect to, say, 216.239.41.104 it gets
> > redirected to the port 8080 on 10.1.0.1
> 
>   Yes, in theory. To trigger the bug, you should connect _from_ the
> gateway to itself. (Note that the ruleset is somewhat flawed, since it
> should redirect only traffic coming in through the LAN interface.) I
> used netcat to connect:
> 
> # nc 10.1.0.1 80
> 
> > but, from 10.1.0.2, if you try to connect directly to port 80 on
> > 10.1.0.1, with your setup where nothing is listening on 80, the box
> > would hang. am i right?!
> 
>   It does not matter if there is something listening on port 80. I think
> it wasn't clear from my previous mail that you have to connect _from_
> the gateway, and not from the attached 10.1.0.0/16 LAN.

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jozsef Kadlecsik]

Hi Jonathan,

On Thu, 11 Mar 2004, Jonathan Cooper wrote:

> Specifically, the message:
>
> kernel: LIST_DELETE: ip_conntrack_core.c:302
> &ct->tuplehash[IP_CT_DIR_REPLY]'(d40acea4) not in ip_conntrack_hash[hr].
>
> From doing a bit of googling it appears this problem has been around for
> some time; hopefully my situation will help you track it down.

We had such problem at kernel 2.4.20 when an internal routine changed.

> The machine exhibiting the problem has 2 network cards: an nForce2 card
> (eth1) on the motherboard (connected to external network) and a sis900
> (eth0) which is connected via crossover cable to another computer. This
> connection is masqueraded.

So you use NAT. If you could test the system without using
NAT at all (I mean no NAT related module loaded in), that could probably
help to identify the problem.

> Under the 2.4.23 and 2.4.24 kernels my machine would oops on reboot (just
> after killing syslog) after the message had occurred. This has not (yet)
> happened on 2.4.25. One common feature, that has occurred once under
> 2.4.23 and once under 2.4.25 is that after several occurrences of this
> message the computer froze solid, with no keyboard, mouse or network
> response.
>
> I have experienced no problems of this nature with Debian's 2.4.22
> kernel.

All the kernels are downloaded from ftp.kernel.org?

Did you compile all of them with gcc 3.3.3? Could you recompile the
kernels with gcc 2.95?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jonathan Cooper]

* Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> [2004-03-11 17:28]:
> Hi Jonathan,
> 
> So you use NAT. If you could test the system without using
> NAT at all (I mean no NAT related module loaded in), that could probably
> help to identify the problem.

Aha. If we have just the following ip* modules loaded: ipt_limit,
ipt_REJECT, ipt_LOG, ipt_state, ip_conntrack, iptable_filter, ip_tables
(and the nat table flushed) then I can access pages via the previously
masqueraded interface without problems.

> On Thu, 11 Mar 2004, Jonathan Cooper wrote:
> > Under the 2.4.23 and 2.4.24 kernels my machine would oops on reboot (just
> > after killing syslog) after the message had occurred. This has not (yet)
> > happened on 2.4.25. One common feature, that has occurred once under
> > 2.4.23 and once under 2.4.25 is that after several occurrences of this
> > message the computer froze solid, with no keyboard, mouse or network
> > response.
> >
> > I have experienced no problems of this nature with Debian's 2.4.22
> > kernel.
> 
> All the kernels are downloaded from ftp.kernel.org?

No, sorry. They're all Debian kernel sources. (I did mention that in an
earlier version of the email, but it got eaten when the system froze...)

> Did you compile all of them with gcc 3.3.3? Could you recompile the
> kernels with gcc 2.95?

I can't remember what version was used to compile 2.4.22. It would've
been whatever the standard gcc was in Debian testing on Dec 12th 2003. I
certainly used gcc 3.3.3 for kernel 2.4.25, and I suspect for .23 & .24
as well, although I'm not sure.

I've just tried the Debian kernel 2.4.25 sources compiled with gcc 2.95
and the kernel.org 2.4.25 sources compiled with gcc 3.3.3. The problem
occurs with both.

Jonathan.

-- 
Jonathan Cooper        MSN: msn@jonc.me.uk        jonc.me.uk/

If you can stay calm when all around you is chaos... then you probably haven't
 fully grasped the seriousness of the situation.

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jozsef Kadlecsik]

On Thu, 11 Mar 2004, Jonathan Cooper wrote:

> > So you use NAT. If you could test the system without using
> > NAT at all (I mean no NAT related module loaded in), that could probably
> > help to identify the problem.
>
> Aha. If we have just the following ip* modules loaded: ipt_limit,
> ipt_REJECT, ipt_LOG, ipt_state, ip_conntrack, iptable_filter, ip_tables
> (and the nat table flushed) then I can access pages via the previously
> masqueraded interface without problems.

So it looks like NAT related bug. Please do the following: both in
ip_conntrack_core.c and ip_nat_core.c enable debugging by changing

#if 0
#define DEBUGP printk
#else
#define DEBUGP(format, args...)
#endif

to

#if 1
#define DEBUGP printk
#else
#define DEBUGP(format, args...)
#endif

at the head of the files and add additional debugging to the
clean_from_lists function in ip_conntrack_core.c like this:

static void
clean_from_lists(struct ip_conntrack *ct)
{
        unsigned int ho, hr;

        DEBUGP("clean_from_lists(%p)\n", ct);

	/* The additional debug lines follow: */
	DEBUGP("original tuple: ");
	DUMP_TUPLE(&ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple);
	DEBUGP("reply tuple:    ");
	DUMP_TUPLE(&ct->tuplehash[IP_CT_DIR_REPLY].tuple);

Recompile the kernel modules, start your system and try to trigger the
bug. Then please send me the kernel log messages or post the URL to
download from.

> > Did you compile all of them with gcc 3.3.3? Could you recompile the
> > kernels with gcc 2.95?
>
> I can't remember what version was used to compile 2.4.22. It would've
> been whatever the standard gcc was in Debian testing on Dec 12th 2003. I
> certainly used gcc 3.3.3 for kernel 2.4.25, and I suspect for .23 & .24
> as well, although I'm not sure.
>
> I've just tried the Debian kernel 2.4.25 sources compiled with gcc 2.95
> and the kernel.org 2.4.25 sources compiled with gcc 3.3.3. The problem
> occurs with both.

OK. That direction is then closed.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Henrik Nordstrom]

On Fri, 12 Mar 2004, Jozsef Kadlecsik wrote:

> Recompile the kernel modules, start your system and try to trigger the
> bug. Then please send me the kernel log messages or post the URL to
> download from.

One more hint: If your system freezes dead then connect a serial console
to collect the log messages. For details on how to set up a serial console
see linux/Documentation/serial-console.txt

Regards
Henrik

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Jonathan Cooper]

* Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> [2004-03-12 09:25]:
> Recompile the kernel modules, start your system and try to trigger the
> bug. Then please send me the kernel log messages or post the URL to
> download from.

I've uploaded snippets of /var/log/kern.log to
<http://jonc.me.uk/stuff/kern.log.bz2> and
<http://jonc.me.uk/stuff/kern_fullish.log.bz2>.

Unfortunately the college network sees a lot of nmb traffic and the like,
so there is a lot of noise in the logs. The former is cut so it just
covers the time period of my second trial.

I tried to fetch a page at about 10.04.01. The connection timed out at
about 10.07.10, and the LIST_DELETE message appears at 10.07.34. The IP
address of my machine on the private (masqueraded) network is
192.168.1.24.

Jonathan.

-- 
Jonathan Cooper        MSN: msn@jonc.me.uk        jonc.me.uk/

Clones are people two

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[wanghtb]

And I think the compiling direction 
 "#ifndef CONFIG_IP_NF_NAT_LOCAL"
maybe not need, since in any situation, we should not 
use ip_conntrack_reply() to change a confirmed ct, which 
will make ct's hash value changed,  and make LIST_DELETE() dump a 
warning: this entry is not in the correct bucket list :-)

>  I think the patch is buggy. The check should be
>
> if (!(info->initialized & (1 << maniptype))
>    && !(ct->status & IPS_CONFIRMED))
>
>instead of
>
>  if (!(info->initialized & (1 << maniptype))
>      && ct->status & IPS_CONFIRMED)
>
>  Fixed patch attached. I've tested it on my UML testbed with stock
> 2.4.24, and seems to solve the problem.
>
> -- 
> Regards,
>   Krisztian KOVACS
>
>--=-+pqSK1ygFZXfde1OZN5T
>Content-Disposition: attachment; filename=renat_confirmed.patch
>Content-Type: text/plain; name=renat_confirmed.patch; charset=iso-8859-2
>Content-Transfer-Encoding: 7bit
>
>--- ip_nat_standalone.c.orig            Fri Nov 28 19:26:21 2003
>+++ ip_nat_standalone.c                 Thu Mar 25 16:43:29 2004
>@@ -114,7 +114,16 @@
>                                WRITE_LOCK(&ip_nat_lock);
>                                /* Seen it before?  This can happen for 
loopback, retrans,
>                                   or local packets.. */
>-                               if (!(info->initialized & (1 << 
maniptype))) {
>+                               if (!(info->initialized & (1 << 
maniptype))
>+#ifndef CONFIG_IP_NF_NAT_LOCAL
>+                                   /* If this session has already been 
confirmed we must not
>+                                    * touch it again even if there is no 
mapping set up.
>+                                    * Can only happen on local->local 
traffic with
>+                                    * CONFIG_IP_NF_NAT_LOCAL disabled.
>+                                    */
>+                                   && !(ct->status & IPS_CONFIRMED)
>+#endif
>+                                   ) {
>                                                unsigned int ret;
> 
>                                                if (ct->master
>
>--=-+pqSK1ygFZXfde1OZN5T--

Re: More on LIST_DELETE message with kernels 2.4.23 through 2.4.25

[Henrik Nordstrom]

On Thu, 1 Apr 2004 wanghtb@legend.com.cn wrote:

> And I think the compiling direction 
>  "#ifndef CONFIG_IP_NF_NAT_LOCAL"
> maybe not need, since in any situation, we should not 
> use ip_conntrack_reply() to change a confirmed ct, which 
> will make ct's hash value changed,  and make LIST_DELETE() dump a 
> warning: this entry is not in the correct bucket list :-)

This can only happen if CONFIG_IP_NF_NAT_LOCAL is not defined.

If it is defined then NAT properly registers mappings in all directions,
and the first (original) part of the if statement makes sure the packet is
not NAT:ed again on local->local traffic.

Regards
Henrik