Bug with netfilter and NFS server on same machine (fwd)

Bug with netfilter and NFS server on same machine (fwd)

[William Stearns]

Good day, Lars,
	Whether or not the netfilter list was able to find an answer, it 
might still make sense to CC that list.
	Cheers,
	- Bill

---------------------------------------------------------------------------
        > It just DOES NOT BELONG in to the kernel-space.
        "People who start using capital letters always seem to have
emotional rather than logical reasons for their argument."
        -- Alan Cox <alan@lxorguk.ukuu.org.uk>
--------------------------------------------------------------------------
William Stearns (wstearns@pobox.com).  Mason, Buildkernel, named2hosts, 
and ipfwadm2ipchains are at:                        http://www.stearns.org
--------------------------------------------------------------------------


---------- Forwarded message ----------
Date: Sat, 23 Nov 2002 19:37:34 +0100
From: Lars Knudsen <gandalfit@virgilio.it>
To: linux-kernel@vger.kernel.org
Subject: Bug with netfilter and NFS server on same machine

I have been experiencing problems running a nfs server and iptables on 
the same machine.The problem was also reported almost a year ago by Paul 
Raines on the netfilter mailing list 
http://lists.netfilter.org/pipermail/netfilter/2002-January/030002.html 
but it seems no solution has been found yet.

The problem is this: A machine running linux 2.4.18 or 2.4.19 works just 
fine when running just the kernel nfsd. A single client connected to the 
server with 100Mbit ethernet sees throughput of 5-10MByte/sec even after 
an hour or two of continous transfers. If the nfs server is also running 
iptables the throughput is initially the same (5-10MByte/sec) but after 
a while (200MByte-500MByte total transfer) the client starts reporting 
"nfs server not responding" followed after a while by "nfs server OK" 
and of course the transfer rate goes way down (< 1MByte/sec). Using 
tcpdump on the client seems to indicate that some packets have their 
headers garbled - wrong fragment ids being the typical error.

Having iptables compiled as modules and simply loading or unloading the 
ipt_conntrack module is
sufficient for causing/removing the problem. Having iptables support 
compiled into the kernel causes the problem allways.

The problem has been verified on 4 different machines with a variety of 
different ethernet cards. In
all cases the network continues to work without problems for all other 
types of traffic - i.e a telnet connection from client to server works 
with no delay and a ftp transfer goes at >5MByte/sec even when nfs 
throughput is suffering.

\Lars Knudsen

Re: Bug with netfilter and NFS server on same machine (fwd)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1786 bytes --]

On Sat, Nov 23, 2002 at 01:44:45PM -0500, William Stearns wrote:
> Good day, Lars,
> 	Whether or not the netfilter list was able to find an answer, it 
> might still make sense to CC that list.

yup, definitely.

> The problem is this: A machine running linux 2.4.18 or 2.4.19 works just 
> fine when running just the kernel nfsd. A single client connected to the 
> server with 100Mbit ethernet sees throughput of 5-10MByte/sec even after 
> an hour or two of continous transfers. If the nfs server is also running 
> iptables the throughput is initially the same (5-10MByte/sec) but after 
> a while (200MByte-500MByte total transfer) the client starts reporting 
> "nfs server not responding" followed after a while by "nfs server OK" 
> and of course the transfer rate goes way down (< 1MByte/sec). Using 
> tcpdump on the client seems to indicate that some packets have their 
> headers garbled - wrong fragment ids being the typical error.

this sounds like a problem with nfs's exceptionally large packet size
(up to 8k) and the resulting udp fragments, which need to get
defragmented and refragmented while conntrack is done.

Apart from that I don't have any idea.  I run lots of linux nfs servers
on machines which have conntrack running, without ever encountering the
problem you are describing.  There has to be something special about
your setup(s) we are missing here.

Any idea?

Thanks.

> \Lars Knudsen

-- 
Live long and prosper
- Harald Welte / laforge@gnumonks.org               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M- 
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Bug with netfilter and NFS server on same machine (fwd)

[Patrick McHardy]

Hi,

Harald Welte wrote:

>On Sat, Nov 23, 2002 at 01:44:45PM -0500, William Stearns wrote:
>  
>
>>Good day, Lars,
>>	Whether or not the netfilter list was able to find an answer, it 
>>might still make sense to CC that list.
>>    
>>
>
>yup, definitely.
>
>  
>
>>The problem is this: A machine running linux 2.4.18 or 2.4.19 works just 
>>fine when running just the kernel nfsd. A single client connected to the 
>>server with 100Mbit ethernet sees throughput of 5-10MByte/sec even after 
>>an hour or two of continous transfers. If the nfs server is also running 
>>iptables the throughput is initially the same (5-10MByte/sec) but after 
>>a while (200MByte-500MByte total transfer) the client starts reporting 
>>"nfs server not responding" followed after a while by "nfs server OK" 
>>and of course the transfer rate goes way down (< 1MByte/sec). Using 
>>tcpdump on the client seems to indicate that some packets have their 
>>headers garbled - wrong fragment ids being the typical error.
>>    
>>
>
>this sounds like a problem with nfs's exceptionally large packet size
>(up to 8k) and the resulting udp fragments, which need to get
>defragmented and refragmented while conntrack is done.
>
I experienced the same problem since almost 6 months with nfs and netfilter,
nfs was veery slow, it wasn't even possible to listen to mp3s over nfs.
removing the conntrack module made it work normal again.
the client ringbuffers fills with
UDP: short packet: 192.168.0.1:55258 58191/236 to 192.168.0.23:55789
UDP: short packet: 192.168.0.1:12966 57456/236 to 192.168.0.23:1590
UDP: short packet: 192.168.0.1:41383 20796/236 to 192.168.0.23:32904
while conntrack module is loaded.
i placed a lot of printk's in conntrack and ip_fragment but couldn find 
any pointers
where the problem might be. interesting might be that changing the mtu 
of the
interface to 1486 works well, even with conntrack loaded .. while 
without conntrack 1500 is ok.

>
>Apart from that I don't have any idea.  I run lots of linux nfs servers
>on machines which have conntrack running, without ever encountering the
>problem you are describing.  There has to be something special about
>your setup(s) we are missing here.
>
>Any idea?
>
>Thanks.
>
>  
>
>>\Lars Knudsen
>>    
>>
>
>  
>
bye,
patrick

Re: Bug with netfilter and NFS server on same machine (fwd)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

On Fri, Dec 06, 2002 at 01:56:18AM +0100, Patrick McHardy wrote:
> Hi,
> 
> Harald Welte wrote:

> I experienced the same problem since almost 6 months with nfs and netfilter,
> nfs was veery slow, it wasn't even possible to listen to mp3s over nfs.

I have experienced the problem the last two days, which I was running a
debian woody system with an 2.4.18-k7 kernel.  I've now compiled my own
kernel again (2.4.20-final with lots of patch-o-matic stuff) and the
problem is gone again [nothing but the kernel on the conntracking
nfs-serve was changed].

I'm running lots of Conectiva and Redhat systems with 2.4.18+ kernels
and don't have this particular conntrack problem on any machine.

> bye,
> patrick

-- 
- Harald Welte / laforge@gnumonks.org               http://www.gnumonks.org/
============================================================================
"If this were a dictatorship, it'd be a heck of a lot easier, just so long
 as I'm the dictator."  --  George W. Bush Dec 18, 2000

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Bug with netfilter and NFS server on same machine (fwd)

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 4897 bytes --]

Hi Harald,

Harald Welte wrote:

>On Fri, Dec 06, 2002 at 01:56:18AM +0100, Patrick McHardy wrote:
>  
>
>>Hi,
>>
>>Harald Welte wrote:
>>    
>>
>>I experienced the same problem since almost 6 months with nfs and netfilter,
>>nfs was veery slow, it wasn't even possible to listen to mp3s over nfs.
>>    
>>
>
>I have experienced the problem the last two days, which I was running a
>debian woody system with an 2.4.18-k7 kernel.  I've now compiled my own
>kernel again (2.4.20-final with lots of patch-o-matic stuff) and the
>problem is gone again [nothing but the kernel on the conntracking
>nfs-serve was changed].
>
The machine making troubles here only runs self-compiled kernels.
IIRC it started with 2.4.18, also lots of patch-o-matic stuff applied.
Since then i've tried a lot kernels, with- and without pom,
atm i'm running 2.4.19-rc4-ac1, attached is a list of applied pom-patches.
If you send me a list of your patches i can try if it helps here.
Also if you would like to have a look for yourself i could create an 
account for you ...

>I'm running lots of Conectiva and Redhat systems with 2.4.18+ kernels
>and don't have this particular conntrack problem on any machine.
>
Here is how tcpdump looks with nfs-server(+conntrack) mtu set to 1500:

0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16283:1480@4440+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16284:1480@2960+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16284:1480@4440+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16284:1480@5920+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 962: 192.168.0.1 > 192.168.0.23: 
(frag 16284:928@7400) (ttl 64, len 948)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 154: 192.168.0.23.359760184 > 
192.168.0.1.2049: 112 read [|nfs] (DF) (ttl 64, id 0, len 140)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1.2049 > 
192.168.0.23.359760184: reply ok 1472 read REG 100644 ids 0/0 [|nfs] 
(frag 16285:1480@0+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16284:1480@4440+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16285:1480@2960+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16285:1480@4440+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1514: 192.168.0.1 > 192.168.0.23: 
(frag 16285:1480@5920+) (ttl 64, len 1500)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 962: 192.168.0.1 > 192.168.0.23: 
(frag 16285:928@7400) (ttl 64, len 948)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18135, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18136, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18137, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18138, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18139, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18140, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18141, len 576)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 590: 192.168.0.23 > 192.168.0.1: 
icmp: ip reassembly time exceeded [tos 0xc0]  (ttl 64, id 18142, len 576)
...

with mtu 1486 tcpdump looks like this (no icmp reassemly time exceeded):
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1498: 192.168.0.1 > 192.168.0.23: 
(frag 16436:1464@1464+) (ttl 64, len 1484)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1498: 192.168.0.1 > 192.168.0.23: 
(frag 16436:1464@2928+) (ttl 64, len 1484)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1498: 192.168.0.1 > 192.168.0.23: 
(frag 16436:1464@4392+) (ttl 64, len 1484)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1498: 192.168.0.1 > 192.168.0.23: 
(frag 16436:1464@5856+) (ttl 64, len 1484)
0:e0:7d:74:ab:cd 0:e0:7d:74:ab:cc 0800 154: 192.168.0.23.2339471672 > 
192.168.0.1.2049: 112 read [|nfs] (DF) (ttl 64, id 0, len 140)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1042: 192.168.0.1 > 192.168.0.23: 
(frag 16436:1008@7320) (ttl 64, len 1028)
0:e0:7d:74:ab:cc 0:e0:7d:74:ab:cd 0800 1498: 192.168.0.1.2049 > 
192.168.0.23.2339471672: reply ok 1456 read REG 100644 ids 0/0 [|nfs] 
(frag 16437:1464@0+) (ttl 64, len 1484)
...

Bye,
Patrick

[-- Attachment #2: ipt_patch_list --]
[-- Type: text/plain, Size: 3609 bytes --]

fuzzy.patch ALREADY APPLIED (0 rejects out of 2 hunks).
iplimit.patch ALREADY APPLIED (0 rejects out of 2 hunks).
ipt_unclean-ubit.patch ALREADY APPLIED (0 rejects out of 1 hunks).
ipv4options.patch ALREADY APPLIED (0 rejects out of 2 hunks).
NETMAP.patch ALREADY APPLIED (0 rejects out of 1 hunks).
nth.patch ALREADY APPLIED (0 rejects out of 2 hunks).
pool.patch ALREADY APPLIED (0 rejects out of 5 hunks).
quota.patch ALREADY APPLIED (0 rejects out of 2 hunks).
random.patch ALREADY APPLIED (0 rejects out of 2 hunks).
condition.patch ALREADY APPLIED (0 rejects out of 2 hunks).
CONNMARK.patch ALREADY APPLIED (0 rejects out of 7 hunks).
h323-conntrack-nat.patch ALREADY APPLIED (0 rejects out of 3 hunks).
ip_tables-proc.patch ALREADY APPLIED (0 rejects out of 5 hunks).
ROUTE.patch ALREADY APPLIED (0 rejects out of 2 hunks).
string.patch ALREADY APPLIED (0 rejects out of 2 hunks).
CONNMARK.patch ALREADY APPLIED (1 rejects out of 7 hunks).
ip_conntrack_find.patch ALREADY APPLIED (0 rejects out of 1 hunks).
ip_ct_refresh_optimization.patch ALREADY APPLIED (3 rejects out of 7 hunks).
newnat-udp-helper.patch ALREADY APPLIED (0 rejects out of 8 hunks).
2.4.18.patch ALREADY APPLIED (5 rejects out of 63 hunks).
ahesp-static.patch ALREADY APPLIED (0 rejects out of 1 hunks).
arptables.patch ALREADY APPLIED (1 rejects out of 4 hunks).
conntrack+nat-helper-unregister.patch ALREADY APPLIED (1 rejects out of 6 hunks).
conntrack.patch ALREADY APPLIED (0 rejects out of 2 hunks).
dscp.patch ALREADY APPLIED (0 rejects out of 2 hunks).
DSCP.patch ALREADY APPLIED (0 rejects out of 2 hunks).
ecn.patch ALREADY APPLIED (0 rejects out of 2 hunks).
ECN.patch ALREADY APPLIED (0 rejects out of 2 hunks).
helper.patch ALREADY APPLIED (0 rejects out of 2 hunks).
ip6tables-export-symbols.patch ALREADY APPLIED (1 rejects out of 2 hunks).
ip_conntrack_protocol_unregister.patch ALREADY APPLIED (2 rejects out of 5 hunks).
ip_nat_irc-srcaddr-fix.patch ALREADY APPLIED (2 rejects out of 4 hunks).
ipt_MIRROR-ttl.patch ALREADY APPLIED (0 rejects out of 3 hunks).
ipt_REJECT-checkentry.patch ALREADY APPLIED (0 rejects out of 1 hunks).
ipt_unclean-ecn.patch ALREADY APPLIED (0 rejects out of 2 hunks).
irc-dcc-mask.patch ALREADY APPLIED (1 rejects out of 2 hunks).
local-nat.patch ALREADY APPLIED (0 rejects out of 13 hunks).
macro-trailing-semicolon-fix.patch ALREADY APPLIED (0 rejects out of 4 hunks).
mangle5hooks.patch ALREADY APPLIED (0 rejects out of 16 hunks).
nat-export_symbols.patch ALREADY APPLIED (1 rejects out of 2 hunks).
nat-memoryleak-fix.patch ALREADY APPLIED (0 rejects out of 1 hunks).
netfilter-arp.patch ALREADY APPLIED (1 rejects out of 5 hunks).
ownercmd.patch ALREADY APPLIED (0 rejects out of 3 hunks).
pkttype.patch ALREADY APPLIED (0 rejects out of 2 hunks).
REJECT-dont_fragment.patch ALREADY APPLIED (0 rejects out of 1 hunks).
REJECT_mark.patch ALREADY APPLIED (0 rejects out of 1 hunks).
skb_clone_copy.patch ALREADY APPLIED (1 rejects out of 5 hunks).
TOS-oops-fix.patch ALREADY APPLIED (0 rejects out of 1 hunks).
ulog-module-unload.patch ALREADY APPLIED (1 rejects out of 2 hunks).
ulog-nlgroup-shift-fix.patch ALREADY APPLIED (1 rejects out of 11 hunks).
ulog-sparc-bitops-fix.patch ALREADY APPLIED (0 rejects out of 1 hunks).
unclean-udpchecksum.patch ALREADY APPLIED (0 rejects out of 2 hunks).
z-newnat16.patch ALREADY APPLIED (9 rejects out of 110 hunks).
z-newnat_assertfix.patch ALREADY APPLIED (0 rejects out of 7 hunks).
z-newnat_changeexpect-lockfix.patch ALREADY APPLIED (0 rejects out of 2 hunks).
ipt_REJECT-fake-source.patch ALREADY APPLIED (0 rejects out of 7 hunks).