Connection problems on large high speed connections.

Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

My firewall has started to drop large connections, like downloading a
>1MB file over FTP or HTTP typically fails. But, it seems that the speed
needs to be over 4-500 K/s before the error occurs. 

I live in Norway and if I ftp from ftp.sunet.se a linux distro ISO for
instance this will faill at about 1 MB size, then it will retry,
continue another megabyte and a new stall. But if I download a large
file from a slow server at about 100-200 K/s the download will continue.

When I flush my iptables script the error is gone. 

I did some tests like remove all iptables entries with -m limit and
such. Also I tested from a nat'ed machine behind the firewall and from
the firewall itself. Same error on both. I also run Snort on the
computer, but it does no difference if it is started or not.

The only thing I can think of is that I not very long ago upgraded from
a 2.4 kernel to a 2.6 kernel. The last two kernels I tried was 2.6.11
and now the 2.6.12-rc3, both produces the same error. I also now
upgraded iptables from 1.2.11 to 1.3.1 but the same error appears. 

My dmesg shows no error messages. How can I get a log from what is
happening? It is not in the FORWARD or OUTPUT chains since it happens
from both internal clients and the firewall itself. Can it be NAT? I use
SNAT to do natting of all connections. How can I debug nat?

I did a ping -f to my gateway, no packet loss, even if i crank the size
up to 1450. I am outta ideas. 

System info:

Fujitsu Server
eepro100 NICs (2)
SCSI disks 2 at 10GB each
Kernel 2.6.11 and 2.6.12-rc3
iptables 1.2.11 and 1.3.1

Hope you have some ideas on my problem. 

Best regards 
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Taylor, Grant]

> When I flush my iptables script the error is gone. 

You say that when you flush your iptables script the error goes away?  Are you flushing the firewall completely or just reapplying / rerunning your firewall script?  What are you doing when you flush the script?



Grant. . . .

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

> ons, 27,.04.2005 kl. 00.03 -0500, skrev Taylor, Grant:
> > When I flush my iptables script the error is gone. 
> 
> You say that when you flush your iptables script the error goes away?  Are you flushing the firewall completely or just reapplying / rerunning your firewall script?  What are you doing when you flush the script?
> 
> 

iptables -F INPUT
iptables -F OUTPUT
iptables -F FORWARD
iptables -F DMZ
iptables -X DMZ
iptables -F ICMP
iptables -X ICMP
iptables -F SMTPDROP
iptables -X SMTPDROP
iptables -t nat -F POSTROUTING # slå av NAT'ing
iptables -t nat -F PREROUTING # slå av redirect

This is what I do when I run my script with stop parameter. The reason I
empty each separately is that I have one chain that I don't want to
reload every time (called NORDIC that is the IP classes of the nordic
countries). When running these -F and -X the problem goes away. 

My firewall script works shortly by masking out all traffic to a x/30
mask that is sent to the DMZ chain. All SMTP (dest. port 25) is filtered
in SMTPDROP, and ICMP in the ICMP chain. 

I run the following kernel modules:

modprobe ip_tables
modprobe ip_conntrack
modprobe ip_conntrack_ftp
modprobe ip_conntrack_irc
modprobe ip_nat_ftp
modprobe ip_nat_irc
modprobe ipt_ULOG

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Jozsef Kadlecsik]

On Wed, 27 Apr 2005, Stian B. Barmen wrote:

> My firewall has started to drop large connections, like downloading a
> >1MB file over FTP or HTTP typically fails. But, it seems that the speed
> needs to be over 4-500 K/s before the error occurs.
[...]
> The only thing I can think of is that I not very long ago upgraded from
> a 2.4 kernel to a 2.6 kernel. The last two kernels I tried was 2.6.11
> and now the 2.6.12-rc3, both produces the same error.

NAT sequence number adjustment is broken since 2.6.11. I believe
the patch was submitted in after 2.6.12-rc3 had been released.

But it's an FTP/IRC/etc data channel related bug. It has nothing to do
with plain HTTP. Do you mean HTTP transfer is also broken or it's just an
FTP transfer initiated by a web client?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

> NAT sequence number adjustment is broken since 2.6.11. I believe
> the patch was submitted in after 2.6.12-rc3 had been released.
> 
> But it's an FTP/IRC/etc data channel related bug. It has nothing to do
> with plain HTTP. Do you mean HTTP transfer is also broken or it's just an
> FTP transfer initiated by a web client?
> 

I get the same error if I run http or ftp. No difference.

Here is wget screenshot:

--09:08:01--
http://www.cpan.no/ports/win32/Standard/x86/perl5.00402-bindist04-bc.zip
  (try: 3) => `perl5.00402-bindist04-bc.zip'
Connecting to www.cpan.no[158.36.2.10]:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 6,176,370 (5,278,602 to go) [application/zip]

28%
[++++++++++++==========>                                                             ] 1,754,080    906.44K/s

09:08:02 (904.70 KB/s) - Read error at byte 1754080/6176370 (Connection
reset by peer). Retrying.

You can see that it downloaded a bit, stopped, resumed, stopped again. 

I hate peer sometimes .. :)

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

Here is another strange thing. When I run lsmod I get this output:

Module                  Size  Used by
ipt_state               1344  -
ipt_REJECT              5376  -
iptable_filter          2048  -
ipt_ULOG                6276  -
ip_nat_irc              1632  -
ip_nat_ftp              2336  -
iptable_nat            19804  -
ip_conntrack_irc       70608  -
ip_conntrack_ftp       71504  -
ip_conntrack           39016  -
ppp_mppe_mppc          14724  -
ppp_generic            21396  -
slhc                    6304  -
ipt_ipp2p               7872  -
ip_tables              17632  -
e100                   30656  -

On my other firewalls I get more in the Used by column. More like this:

ipt_limit                984   6  (autoclean)
ipt_LOG                 3448   1  (autoclean)
ipt_REJECT              3288  15  (autoclean)
ipt_state                568   5  (autoclean)
iptable_filter          1740   1  (autoclean)
ip_nat_irc              2704   0  (unused)
ip_nat_ftp              3440   0  (unused)
ip_conntrack_irc        3344   1
ip_conntrack_ftp        4496   1
iptable_nat            18232   3  [ip_nat_irc ip_nat_ftp]
ip_conntrack           23048   4  [ipt_state ip_nat_irc ip_nat_ftp
ip_conntrack_irc ip_conntrack_ftp iptable_nat]
ip_tables              12992   8  [ipt_limit ipt_LOG ipt_REJECT
ipt_state iptable_filter iptable_nat]
smbfs                  39984   1  (autoclean)
ppp_mppe               22680   0
eepro100               20180   2
mii                     2544   0  [eepro100]
(this is a screen shot from a similar server but running 2.4 kernel)

Does this have anything to do with my file transfer problem?

Best regards 
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

I saw this on the netfilter page. Maybe I forgot to say I have a dual
cpu system. Can this be the source of my problems?


Author: Patrick McHardy, <kaber@trash.net> 
Status: Working

This patch fixes a deadlock condition in conntrack/nat-helpers

There is a possible deadlock condition with conntrack/nat-helpers:

CPU1:
conntrack-helper:help: lock(private_lock)
ip_conntrack_expect_related: write_lock(ip_conntrack_lock)

CPU2:
nat-core:do_bindings: read_lock(ip_conntrack_lock)
nat-helper:help: lock(private_lock)

The lock in the nat-helper is unneccessary because the expectation
is never changed and is protected by ip_conntrack_lock.


How do I go about to implement this patch? I cannot find it patchomatic-ng, maybe I am doing it wrongly. 

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

Solved it! :)

Or rather, a friend of mine assisted me and we found the trouble. 

In the code I added at the end of INPUT, FORWARD and the redirected DMZ
chain the following:

iptables -A INPUT -p tcp -j REJECT --reject-with tcp-reset
iptables -A FORWARD -p tcp -j REJECT --reject-with tcp-reset
iptables -A DMZ -p tcp -j REJECT --reject-with tcp-reset


I removed the --reject-with tcp-reset on each line and the problem
dissapeard.

The strange thing is that this communication should never reach this
rule. When the communcation is established it should hit the rule:

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT 

Should it not? (this rule runs before the -j DMZ and I have another one
for INPUT).

I have no explanation for this behaviour. Will try to log and see what I
can find but for now this is all I know. 

Thanks for the replies so far. 

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Jozsef Kadlecsik]

On Wed, 27 Apr 2005, Stian B. Barmen wrote:

> In the code I added at the end of INPUT, FORWARD and the redirected DMZ
> chain the following:
>
> iptables -A INPUT -p tcp -j REJECT --reject-with tcp-reset
> iptables -A FORWARD -p tcp -j REJECT --reject-with tcp-reset
> iptables -A DMZ -p tcp -j REJECT --reject-with tcp-reset
>
> I removed the --reject-with tcp-reset on each line and the problem
> dissapeard.
>
> The strange thing is that this communication should never reach this
> rule. When the communcation is established it should hit the rule:
>
> -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
>
> Should it not? (this rule runs before the -j DMZ and I have another one
> for INPUT).

Then there were packets flagged as INVALID by conntrack, which are of
course not matched by the states above. The reject line however matched
them and dutifully generated the RST segment, which tore down the
connection.

> I have no explanation for this behaviour. Will try to log and see what I
> can find but for now this is all I know.

Enable logging invalid packets by

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid

and make sure ipt_LOG is loaded in.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

> Then there were packets flagged as INVALID by conntrack, which are of
> course not matched by the states above. The reject line however matched
> them and dutifully generated the RST segment, which tore down the
> connection.

But what is the reason for the difference in behaviour for -j REJECT vs
-j RECECT --reject-with tcp-reset? Why does one kill the connection and
not the other?

> Enable logging invalid packets by
> 
> echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
> 
> and make sure ipt_LOG is loaded in.

Will do this :)

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Jozsef Kadlecsik]

On Wed, 27 Apr 2005, Stian B. Barmen wrote:

> > Then there were packets flagged as INVALID by conntrack, which are of
> > course not matched by the states above. The reject line however matched
> > them and dutifully generated the RST segment, which tore down the
> > connection.
>
> But what is the reason for the difference in behaviour for -j REJECT vs
> -j RECECT --reject-with tcp-reset? Why does one kill the connection and
> not the other?

A "-j RECECT --reject-with tcp-reset" generates a TCP RST, which always
kills the connection. A "-j RECECT" generates an ICMP error message, which
- depending on the OS which receives the ICMP packet - might terminate a
TCP connection or might not. That is the very reason why "--reject-with
tcp-reset" is required.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]


> A "-j RECECT --reject-with tcp-reset" generates a TCP RST, which always
> kills the connection. A "-j RECECT" generates an ICMP error message, which
> - depending on the OS which receives the ICMP packet - might terminate a
> TCP connection or might not. That is the very reason why "--reject-with
> tcp-reset" is required.

Thanks for that info! :)

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

Just to satisfy my couriosity, I logged in ulog the invalig packets, and
there are quite a few. How many should I expect to see and count as
normal? 

For instance I downloaded a file from a reasonably fast FTP server at
about 7 MB, and during I logged three invalid TCP packets. 

Apr 28 22:07:25 fire Invalid:  IN=eth1 OUT=
MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=47468 CE DF
PROTO=TCP SPT=80 DPT=33553 SEQ=985943197 ACK=497088462 WINDOW=6432 ACK
URGP=0
Apr 28 22:07:43 fire Invalid:  IN=eth1 OUT=
MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=52274 CE DF
PROTO=TCP SPT=80 DPT=33553 SEQ=989439897 ACK=497088462 WINDOW=6432 ACK
URGP=0
Apr 28 22:07:47 fire Invalid:  IN=eth1 OUT=
MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=53186 CE DF
PROTO=TCP SPT=80 DPT=33553 SEQ=990104197 ACK=497088462 WINDOW=6432 ACK
URGP=0

Should I just count this as normal? I thougt about using a limit per
second to log if it happened more than 2-3 per second. 

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]

Re: Connection problems on large high speed connections.

[Jason Opperisano]

On Thu, Apr 28, 2005 at 10:16:20PM +0200, Stian B. Barmen wrote:
> Just to satisfy my couriosity, I logged in ulog the invalig packets, and
> there are quite a few. How many should I expect to see and count as
> normal? 
> 
> For instance I downloaded a file from a reasonably fast FTP server at
> about 7 MB, and during I logged three invalid TCP packets. 
> 
> Apr 28 22:07:25 fire Invalid:  IN=eth1 OUT=
> MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
> DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=47468 CE DF
> PROTO=TCP SPT=80 DPT=33553 SEQ=985943197 ACK=497088462 WINDOW=6432 ACK
> URGP=0
> Apr 28 22:07:43 fire Invalid:  IN=eth1 OUT=
> MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
> DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=52274 CE DF
> PROTO=TCP SPT=80 DPT=33553 SEQ=989439897 ACK=497088462 WINDOW=6432 ACK
> URGP=0
> Apr 28 22:07:47 fire Invalid:  IN=eth1 OUT=
> MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
> DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=53186 CE DF
> PROTO=TCP SPT=80 DPT=33553 SEQ=990104197 ACK=497088462 WINDOW=6432 ACK
> URGP=0

the only thing that jumps out at me is that all those packets have the
CE bit set (Congestion Experienced).  care to share with us the rule
that creates those log entries?  is it just "-m state --state INVALID -j
LOG"?  i would be very surprised if setting CE caused a packet to
identified as INVALID...

-j

--
"Stewie: I'll wait until you're finished. Are you done? Because
 I thought this show was called "Kids Say the Darndest Things," not
 "Old Black Comedians Who Never Shut The Hell Up.""
        --Family Guy

Re: Connection problems on large high speed connections.

[Stian B. Barmen]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

> > Apr 28 22:07:47 fire Invalid:  IN=eth1 OUT=
> > MAC=00:d0:b7:1d:cc:7d:00:90:69:f0:b0:20:08:00  SRC=156.56.247.195
> > DST=217.199.xx.18 LEN=1500 TOS=00 PREC=0x00 TTL=53 ID=53186 CE DF
> > PROTO=TCP SPT=80 DPT=33553 SEQ=990104197 ACK=497088462 WINDOW=6432 ACK
> > URGP=0
> 
> the only thing that jumps out at me is that all those packets have the
> CE bit set (Congestion Experienced).  care to share with us the rule
> that creates those log entries?  is it just "-m state --state INVALID -j
> LOG"?  i would be very surprised if setting CE caused a packet to
> identified as INVALID...
> 

Yes of course I share willingly :)

        iptables -L INVALIDDROP -n &>/dev/null ||\
	iptables -N INVALIDDROP
        iptables -A INVALIDDROP -j ULOG --ulog-prefix "Invalid: "
        iptables -A INVALIDDROP -j DROP
	iptables -A INPUT -m state --state INVALID -j INVALIDDROP

This is the lines from my firewall script. It is like you say only -m
state --state INVALID that is used. 

From my iptables -L you can see it's right there at the top:

fire root # iptables -L INPUT -n
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  127.0.0.0/8          127.0.0.0/8
INVALIDDROP  all  --  0.0.0.0/0            0.0.0.0/0           state
INVALID
..... <snip> .....

Best regards
Stian B. Barmen

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2685 bytes --]