From: Sven Riedel <sr@securenet.de>
To: Patrick McHardy <kaber@trash.net>
Cc: netfilter@vger.kernel.org,
Netfilter Developer Mailing List
<netfilter-devel@vger.kernel.org>
Subject: Re: Transfer stalls with NAT under 2.6.24.3
Date: Wed, 26 Mar 2008 11:21:13 +0100 [thread overview]
Message-ID: <47EA2399.1080201@securenet.de> (raw)
In-Reply-To: <47EA1653.3080300@trash.net>
Patrick McHardy wrote:
> Sven Riedel wrote:
>> Hi,
>> I've run into a strange problem where large file transfers start
>> stalling over a NATed connection. Packet traces reveal that ACK
>> packets are sometimes not being passed through to the inside (NATed)
>> host, which results in a transfer stall until a tcp timeout occurrs
>> and the other side retransmits the ACK.
>>
>> This only seems to happen if the conntrack table on the firewall
>> already contains an entry for the same source and destination in
>> TIME_WAIT state. If no conntrack entries exist for the same source and
>> destination, the packets flow fine.
>>
>> The problem seems to be alevated by setting ip_conntrac_tcp_be_liberal
>> to 1, but this seems to be only a workaround not a real solution.
>>
>> Scatter gather and tcp segment offloading have been disabled in the
>> relevant NICs on the firewall during debugging, to make sure this
>> isn't a hardware issue.
>>
>> Is this issue known/is there a patch available or would further
>> information be needed to help debug the problem?
>
> 2.6.24.3 includes a patches that was supposed to fix problems
> with connections in TIME_WAIT state. Does 2.6.24.2 work better
> for you?
The firewall system in question is currently productive. I _might_ be
able to try the other kernel tomorrow morning. Once I am able to try it
I'll let you know.
>
> Please enable conntrack logging for TCP by executing:
>
> echo 6 >/proc/sys/net/netfilter/nf_conntrack_log_invalid
>
> and check whether you get any messages in the ring buffer.
Yep, lots ;)
In the following 100.100.100.100 is the external machine and
200.200.200.200 is the NAT IP-Address on the firewall. A 5MB file was
transferred via scp to 100.100.100.100 from the internal network.
The output during a "clean" run, with an empty conntrack table and no
stalls:
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42121
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E351B138AA40101050AE50974FBE5097A53)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42122
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E361B138AA40101050AE50974FBE5097FAB)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42123
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E361B138AA40101050AE50974FBE5098503)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42124
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E371B138AA40101050AE50974FBE5098A5B)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42125
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E381B138AA40101050AE50974FBE5098FB3)
printk: 24 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42248
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720852 ACK=3828837755 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A45585F911B138E140101050AE50FB2C3E50FD2D3)
printk: 31 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42465
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355721284 ACK=3829614779 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455861861B1392E10101050AE51B935BE51BB8C3)
printk: 25 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42718
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355721716 ACK=3830353499 WINDOW=42408
RES=0x00 ACK URGP=0 OPT (0101080A455863DA1B1398B70101050AE526E3ABE526E903)
printk: 57 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42976
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355722052 ACK=3830954051 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455865791B139CBE0101050AE52FFD8BE530284B)
printk: 27 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=72 TOS=0x00 PREC=0x00 TTL=56
ID=43306
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355722580 ACK=3831787163 WINDOW=42408
RES=0x00 ACK URGP=0 OPT
(0101080A455867731B13A19501010512E53CCB53E53CD653E53CBE93E53CC3EB)
printk: 74 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=43789
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355723252 ACK=3832978011 WINDOW=42408
RES=0x00 ACK URGP=0 OPT (0101080A45586A571B13A8CF0101050AE54EDEABE54EE403)
During a run with stalls:
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=80 TOS=0x00 PREC=0x00 TTL=56
ID=44105
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160349927 ACK=596614326 WINDOW=49248
RES=0x00 ACK URGP=0 OPT
(0101080A4558793C1B13CE350101051A491E8751491E8CA9491E7B71491E81F9491E40A9491E5B61)
^^^^ Transfer stalled here for ~10 seconds.
printk: 22 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=72 TOS=0x00 PREC=0x00 TTL=56
ID=44113
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160349927 ACK=596632110 WINDOW=49248
RES=0x00 ACK URGP=0 OPT
(0101080A45587D301B13D81801010512491E8751491E8CA9491E7B71491E81F9)
printk: 12 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=44114
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160349927 ACK=596635150 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455881B21B13E35A0101050A491E8751491E8CA9)
printk: 14 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=7320
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160350311 ACK=597280038 WINDOW=27360
RES=0x00 ACK URGP=0 OPT (0101080A455883D31B13E8820101050A49286E71492873C9)
printk: 32 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=7451
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160350503 ACK=597578342 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455885161B13EBD30101050A492CEBA9492CF659)
printk: 35 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=7786
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160350983 ACK=598415558 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455887081B13F0890101050A4939B2094939E221)
printk: 54 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8021
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160351319 ACK=598980542 WINDOW=42408
RES=0x00 ACK URGP=0 OPT (0101080A455889151B13F5C00101050A4942510149425659)
printk: 43 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8205
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160351559 ACK=599403254 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A45588B011B13FA9B0101050A4948C4394948C991)
printk: 40 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8531
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160352039 ACK=600218582 WINDOW=45144
RES=0x00 ACK URGP=0 OPT (0101080A45588D371B1400160101050A4955351949553A71)
printk: 49 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8871
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160352519 ACK=601058534 WINDOW=38304
RES=0x00 ACK URGP=0 OPT (0101080A45588F521B1405500101050A4962062949620B81)
printk: 45 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8988
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160352663 ACK=601307510 WINDOW=41040
RES=0x00 ACK URGP=0 OPT (0101080A4558910A1B1409AB0101050A4965D2B94965D811)
Regards,
Sven
--
sven.riedel@securenet.de
SecureNet GmbH
Intranet & Internet Solutions
Frankfurter Ring 193a
D-80807 München
Tel: +49 89 32133-632
Fax: +49 89 32133-699
Zentrale: -600
www.securenet.de
Sitz der Gesellschaft: München
HRB München 118876
Geschäftsführer: Thomas Schreiber
next prev parent reply other threads:[~2008-03-26 10:21 UTC|newest]
Thread overview: 115+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <47EA0DAB.7080205@securenet.de>
2008-03-26 9:24 ` Transfer stalls with NAT under 2.6.24.3 Patrick McHardy
2008-03-26 10:21 ` Sven Riedel [this message]
2008-03-26 15:47 ` Patrick McHardy
2008-03-26 18:45 ` Jozsef Kadlecsik
2008-03-26 19:16 ` Krzysztof Oledzki
2008-03-31 6:53 ` Sven Riedel
2008-07-04 14:54 ` TCP connection stalls under 2.6.24.7 Thomas Jarosch
2008-07-04 20:58 ` Jozsef Kadlecsik
2008-07-04 21:04 ` Jozsef Kadlecsik
2008-07-07 9:18 ` Thomas Jarosch
2008-07-07 13:18 ` Thomas Jarosch
2008-07-10 13:17 ` Jozsef Kadlecsik
2008-07-10 14:12 ` Thomas Jarosch
2008-07-10 21:21 ` Jozsef Kadlecsik
2008-07-11 14:33 ` Thomas Jarosch
2008-07-15 11:47 ` Thomas Jarosch
2008-07-15 16:10 ` Thomas Jarosch
2008-07-15 18:30 ` Dâniel Fraga
2008-07-31 4:47 ` Dâniel Fraga
2008-07-31 7:39 ` Ilpo Järvinen
2008-08-02 12:24 ` Dâniel Fraga
2008-07-15 20:17 ` Ilpo Järvinen
2008-07-16 8:07 ` Thomas Jarosch
2008-07-16 9:03 ` Thomas Jarosch
2008-07-17 13:55 ` Ilpo Järvinen
2008-07-17 15:15 ` Thomas Jarosch
2008-07-17 15:53 ` Ilpo Järvinen
2008-07-18 9:14 ` Thomas Jarosch
2008-07-18 13:55 ` Ilpo Järvinen
2008-07-18 14:02 ` Thomas Jarosch
2008-07-19 7:35 ` Ilpo Järvinen
2008-07-25 10:00 ` Ilpo Järvinen
2008-07-25 13:00 ` Thomas Jarosch
2008-07-25 14:06 ` Ilpo Järvinen
2008-07-25 15:34 ` Thomas Jarosch
2008-07-31 7:39 ` Thomas Jarosch
2008-07-31 12:44 ` Dâniel Fraga
2008-07-31 13:47 ` Thomas Jarosch
2008-07-31 14:11 ` Dâniel Fraga
2008-08-06 18:53 ` Dâniel Fraga
2008-08-07 6:54 ` Ilpo Järvinen
2008-08-07 11:50 ` Denys Fedoryshchenko
2008-08-07 12:11 ` Thomas Jarosch
2008-08-07 12:14 ` Ilpo Järvinen
2008-08-07 12:23 ` Denys Fedoryshchenko
2008-08-08 9:56 ` Ilpo Järvinen
2008-08-08 10:32 ` Denys Fedoryshchenko
2008-08-07 11:33 ` [PATCH] tcp FRTO: in-order-only "TCP proxy" fragility workaround Ilpo Järvinen
2008-08-08 4:42 ` Bill Fink
2008-08-08 10:32 ` Ilpo Järvinen
2008-08-11 21:44 ` David Miller
2008-08-12 7:46 ` Thomas Jarosch
2008-08-12 8:18 ` David Miller
2008-08-12 17:43 ` Dâniel Fraga
2008-08-12 17:52 ` Ilpo Järvinen
2008-08-13 17:53 ` Dâniel Fraga
2008-08-13 18:34 ` Ilpo Järvinen
2008-08-15 4:34 ` Dâniel Fraga
2008-08-15 7:06 ` Ilpo Järvinen
2008-08-15 21:35 ` Dâniel Fraga
2008-08-15 22:06 ` Ilpo Järvinen
2008-08-15 23:57 ` Dâniel Fraga
2008-08-16 2:15 ` Dâniel Fraga
2008-08-16 7:10 ` Ilpo Järvinen
2008-08-16 19:18 ` Ilpo Järvinen
2008-08-17 0:36 ` Dâniel Fraga
2008-08-19 10:38 ` Ilpo Järvinen
2008-08-20 0:34 ` Dâniel Fraga
2008-08-20 7:57 ` Ilpo Järvinen
2008-08-20 12:37 ` Ilpo Järvinen
2008-08-22 21:32 ` Dâniel Fraga
2008-08-22 21:37 ` David Miller
2008-08-23 14:14 ` Dâniel Fraga
2008-08-23 14:38 ` Ilpo Järvinen
2008-08-24 19:38 ` Dâniel Fraga
2008-08-26 14:10 ` Ilpo Järvinen
2008-08-26 14:32 ` Ilpo Järvinen
2008-08-26 17:18 ` Dâniel Fraga
2008-08-26 20:40 ` Ilpo Järvinen
2008-08-26 21:17 ` Dâniel Fraga
2008-08-27 10:22 ` Ilpo Järvinen
2008-08-27 19:51 ` Dâniel Fraga
2008-08-27 20:32 ` Ilpo Järvinen
2008-08-27 20:50 ` Dâniel Fraga
2008-08-27 21:25 ` Ilpo Järvinen
2008-08-27 21:42 ` Dâniel Fraga
2008-08-27 22:24 ` Dâniel Fraga
2008-08-28 21:49 ` Dâniel Fraga
2008-08-29 13:07 ` Ilpo Järvinen
2008-08-29 17:41 ` Dâniel Fraga
2008-09-01 7:11 ` Ilpo Järvinen
2008-08-30 6:56 ` Dâniel Fraga
2008-09-01 7:11 ` Ilpo Järvinen
2008-09-07 8:17 ` Dâniel Fraga
2008-09-08 10:27 ` Ilpo Järvinen
2008-09-08 20:20 ` Dâniel Fraga
2008-09-11 13:44 ` Ilpo Järvinen
2008-09-11 17:30 ` Dâniel Fraga
2008-09-12 10:16 ` Ilpo Järvinen
2008-09-13 23:31 ` Dâniel Fraga
2008-09-16 12:10 ` Ilpo Järvinen
2008-09-16 14:24 ` Dâniel Fraga
2008-09-17 10:23 ` Ilpo Järvinen
2008-09-18 20:35 ` Dâniel Fraga
2008-09-18 21:04 ` Ilpo Järvinen
2008-09-21 3:02 ` Dâniel Fraga
2008-09-22 4:23 ` Dâniel Fraga
2008-09-22 11:22 ` Ilpo Järvinen
2008-09-22 16:13 ` Dâniel Fraga
2008-09-15 19:42 ` Dâniel Fraga
2008-09-11 18:12 ` Dâniel Fraga
2008-08-15 21:59 ` Dâniel Fraga
2008-08-13 8:00 ` Thomas Jarosch
2008-08-22 21:18 ` Ilpo Järvinen
2008-08-11 21:41 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47EA2399.1080201@securenet.de \
--to=sr@securenet.de \
--cc=kaber@trash.net \
--cc=netfilter-devel@vger.kernel.org \
--cc=netfilter@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox