Troubles with NFS & ip_conntrack: packets go to wrong mac

Troubles with NFS & ip_conntrack: packets go to wrong mac

[Patrick McHardy]

I've been experiencing strange problems with nfs and ip_conntrack for a 
while now,
unfortunately noone so far was able to help. The problem occurs when 
ip_conntrack
is loaded on the nfs server. nfs reads hang and the clients start logging

UDP: short packet: 192.168.0.1:0 0/120 to 192.168.0.23:0
UDP: short packet: 192.168.0.1:6439 28562/120 to 192.168.0.23:60558

There are two ways to make it work: 1. remove ip_conntrack or 2. set mtu 
to 1484 on
the nfs server. One suspicion was ip_conntrack breaking udp path mtu 
discovery
since it seems to defragment packets with DF|MF and refragment them 
(with possibly
different mtu) at POSTROUTING. This doesn't seem to be the problem, but 
i noted the
nfs server sends out fragments with wrong destination mac.

This is a packet captured on 192.168.0.23:
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 
192.168.0.223: (frag 44777:1480@4440+) (ttl 64, len 1500, bad cksum 2294!)

This happens every 1-30 seconds. I never saw more than one misdirected 
fragment per packet.

Relevent parts from neighbour table:
192.168.0.223 dev eth0 lladdr 00:e0:29:3c:c1:c9 nud reachable
192.168.0.23 dev eth0 lladdr 00:e0:7d:74:ab:cd nud reachable

On 192.168.0.223 packets for 192.168.0.23 show up. Both clients time out 
during reassembly.
I placed some printks though the netfilter code and ip_output.c but 
couldn't find any further
pointers. I looked for broken checksums (something seems to alter the ip 
after checksumming)
in ip_finish_output2, but everything is ok there. I can't see anything 
netfilter related touching packets after that.
Any help and/or pointers where to look further would be appreciated.

Regards,
Patrick

Re: Troubles with NFS & ip_conntrack: packets go to wrong mac

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Wed, Feb 12, 2003 at 06:05:15PM +0100, Patrick McHardy wrote:
> I've been experiencing strange problems with nfs and ip_conntrack for a 
> while now, unfortunately noone so far was able to help. The problem
> occurs when ip_conntrack is loaded on the nfs server. nfs reads hang
> and the clients start logging

Just for consistency: Can you please report this to
http://bugzilla.netfilter.org/ ? 

Addiditional information like what kind of machine (UP/SMP), and network
board might be interesting.

> Regards,
> Patrick

-- 
- Harald Welte <laforge@gnumonks.org>               http://www.gnumonks.org/
============================================================================
"If this were a dictatorship, it'd be a heck of a lot easier, just so long
 as I'm the dictator."  --  George W. Bush Dec 18, 2000

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Troubles with NFS & ip_conntrack: packets go to wrong mac

[Patrick McHardy]

Some of the information in my last mail was wrong, i'll cut-and-paste 
the information entered into the netfilter
bugtracking system, maybe someone has a suggestion. The important part 
is the packets checksum turns out to
be right if the destination ip is replaced by the ip the mac belongs to, 
so the mac is correct.

--cut-n-paste--

With ip_conntrack loaded on the nfs server, client reads time out.
Clients start logging:

UDP: short packet: 192.168.0.1:0 0/120 to 192.168.0.23:0
UDP: short packet: 192.168.0.1:6439 28562/120 to 192.168.0.23:60558

tcpdump shows corrupted packets:
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.223: (frag
42878:1480@4440+) (ttl 64, len 1500, bad cksum 29ff!)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.223: (frag
42879:1480@4440+) (ttl 64, len 1500, bad cksum 29fe!)

0:e0:7d:74:ab:cd is not the mac of 192.168.0.223 but 192.168.0.23. Both are
nfs-clients. If more nfs-clients show up more incorrect destination ips appear.
The packets checksum is correct if the (incorrect) destination ip is replaced by
the correct destination ip. Only single fragments have incorrect destination,
the remaining fragments of a packet are fine. Packets verified (checksum) in
ip_finish_output2 show no corruption.

Neighbour table of nfs server:
192.168.0.223 dev eth0 lladdr 00:e0:29:3c:c1:c9 nud reachable
192.168.0.23 dev eth0 lladdr 00:e0:7d:74:ab:cd nud reachable

The problem goes away as soon as ip_conntrack is unloaded. Another possibility
is to set the interface mtu to 1486 on the nfs server.

CPU: AMD-K6(tm) 3D processor, 256MB RAM
Kernel: 2.4.21-pre3 with few netfilter patches applied, problem also exists in
vanilla kernel, first noticed around 2.4.17

lspci:
00:00.0 Host bridge: Acer Laboratories Inc. [ALi] M1541 (rev 04)
00:01.0 PCI bridge: Acer Laboratories Inc. [ALi] M5243 (rev 04)
00:07.0 ISA bridge: Acer Laboratories Inc. [ALi] M1533 PCI to ISA Bridge
[Aladdin IV] (rev c3)
00:08.0 VGA compatible unclassified device: S3 Inc. 86c864 [Vision 864 DRAM] vers 0
00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
00:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
00:0f.0 IDE interface: Acer Laboratories Inc. [ALi] M5229 IDE (rev c2)

--end-cut-n-paste--

Bye,
Patrick



Patrick McHardy wrote:

> I've been experiencing strange problems with nfs and ip_conntrack for 
> a while now,
> unfortunately noone so far was able to help. The problem occurs when 
> ip_conntrack
> is loaded on the nfs server. nfs reads hang and the clients start logging
>
> UDP: short packet: 192.168.0.1:0 0/120 to 192.168.0.23:0
> UDP: short packet: 192.168.0.1:6439 28562/120 to 192.168.0.23:60558
>
> There are two ways to make it work: 1. remove ip_conntrack or 2. set 
> mtu to 1484 on
> the nfs server. One suspicion was ip_conntrack breaking udp path mtu 
> discovery
> since it seems to defragment packets with DF|MF and refragment them 
> (with possibly
> different mtu) at POSTROUTING. This doesn't seem to be the problem, 
> but i noted the
> nfs server sends out fragments with wrong destination mac.
>
> This is a packet captured on 192.168.0.23:
> 0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 
> 192.168.0.223: (frag 44777:1480@4440+) (ttl 64, len 1500, bad cksum 
> 2294!)
>
> This happens every 1-30 seconds. I never saw more than one misdirected 
> fragment per packet.
>
> Relevent parts from neighbour table:
> 192.168.0.223 dev eth0 lladdr 00:e0:29:3c:c1:c9 nud reachable
> 192.168.0.23 dev eth0 lladdr 00:e0:7d:74:ab:cd nud reachable
>
> On 192.168.0.223 packets for 192.168.0.23 show up. Both clients time 
> out during reassembly.
> I placed some printks though the netfilter code and ip_output.c but 
> couldn't find any further
> pointers. I looked for broken checksums (something seems to alter the 
> ip after checksumming)
> in ip_finish_output2, but everything is ok there. I can't see anything 
> netfilter related touching packets after that.
> Any help and/or pointers where to look further would be appreciated.
>
> Regards,
> Patrick
>