From: Sven Riedel <sr@securenet.de>
To: Patrick McHardy <kaber@trash.net>
Cc: netfilter@vger.kernel.org,
Netfilter Developer Mailing List
<netfilter-devel@vger.kernel.org>
Subject: Re: Transfer stalls with NAT under 2.6.24.3
Date: Wed, 26 Mar 2008 11:21:13 +0100 [thread overview]
Message-ID: <47EA2399.1080201@securenet.de> (raw)
In-Reply-To: <47EA1653.3080300@trash.net>
Patrick McHardy wrote:
> Sven Riedel wrote:
>> Hi,
>> I've run into a strange problem where large file transfers start
>> stalling over a NATed connection. Packet traces reveal that ACK
>> packets are sometimes not being passed through to the inside (NATed)
>> host, which results in a transfer stall until a tcp timeout occurrs
>> and the other side retransmits the ACK.
>>
>> This only seems to happen if the conntrack table on the firewall
>> already contains an entry for the same source and destination in
>> TIME_WAIT state. If no conntrack entries exist for the same source and
>> destination, the packets flow fine.
>>
>> The problem seems to be alevated by setting ip_conntrac_tcp_be_liberal
>> to 1, but this seems to be only a workaround not a real solution.
>>
>> Scatter gather and tcp segment offloading have been disabled in the
>> relevant NICs on the firewall during debugging, to make sure this
>> isn't a hardware issue.
>>
>> Is this issue known/is there a patch available or would further
>> information be needed to help debug the problem?
>
> 2.6.24.3 includes a patches that was supposed to fix problems
> with connections in TIME_WAIT state. Does 2.6.24.2 work better
> for you?
The firewall system in question is currently productive. I _might_ be
able to try the other kernel tomorrow morning. Once I am able to try it
I'll let you know.
>
> Please enable conntrack logging for TCP by executing:
>
> echo 6 >/proc/sys/net/netfilter/nf_conntrack_log_invalid
>
> and check whether you get any messages in the ring buffer.
Yep, lots ;)
In the following 100.100.100.100 is the external machine and
200.200.200.200 is the NAT IP-Address on the firewall. A 5MB file was
transferred via scp to 100.100.100.100 from the internal network.
The output during a "clean" run, with an empty conntrack table and no
stalls:
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42121
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E351B138AA40101050AE50974FBE5097A53)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42122
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E361B138AA40101050AE50974FBE5097FAB)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42123
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E361B138AA40101050AE50974FBE5098503)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42124
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E371B138AA40101050AE50974FBE5098A5B)
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42125
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720612 ACK=3828427355 WINDOW=47880
RES=0x00 ACK URGP=0 OPT (0101080A45585E381B138AA40101050AE50974FBE5098FB3)
printk: 24 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42248
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355720852 ACK=3828837755 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A45585F911B138E140101050AE50FB2C3E50FD2D3)
printk: 31 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42465
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355721284 ACK=3829614779 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455861861B1392E10101050AE51B935BE51BB8C3)
printk: 25 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42718
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355721716 ACK=3830353499 WINDOW=42408
RES=0x00 ACK URGP=0 OPT (0101080A455863DA1B1398B70101050AE526E3ABE526E903)
printk: 57 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=42976
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355722052 ACK=3830954051 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455865791B139CBE0101050AE52FFD8BE530284B)
printk: 27 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=72 TOS=0x00 PREC=0x00 TTL=56
ID=43306
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355722580 ACK=3831787163 WINDOW=42408
RES=0x00 ACK URGP=0 OPT
(0101080A455867731B13A19501010512E53CCB53E53CD653E53CBE93E53CC3EB)
printk: 74 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=43789
DF PROTO=TCP SPT=22 DPT=43021 SEQ=355723252 ACK=3832978011 WINDOW=42408
RES=0x00 ACK URGP=0 OPT (0101080A45586A571B13A8CF0101050AE54EDEABE54EE403)
During a run with stalls:
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=80 TOS=0x00 PREC=0x00 TTL=56
ID=44105
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160349927 ACK=596614326 WINDOW=49248
RES=0x00 ACK URGP=0 OPT
(0101080A4558793C1B13CE350101051A491E8751491E8CA9491E7B71491E81F9491E40A9491E5B61)
^^^^ Transfer stalled here for ~10 seconds.
printk: 22 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=72 TOS=0x00 PREC=0x00 TTL=56
ID=44113
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160349927 ACK=596632110 WINDOW=49248
RES=0x00 ACK URGP=0 OPT
(0101080A45587D301B13D81801010512491E8751491E8CA9491E7B71491E81F9)
printk: 12 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=44114
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160349927 ACK=596635150 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455881B21B13E35A0101050A491E8751491E8CA9)
printk: 14 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=7320
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160350311 ACK=597280038 WINDOW=27360
RES=0x00 ACK URGP=0 OPT (0101080A455883D31B13E8820101050A49286E71492873C9)
printk: 32 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=7451
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160350503 ACK=597578342 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455885161B13EBD30101050A492CEBA9492CF659)
printk: 35 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=7786
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160350983 ACK=598415558 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A455887081B13F0890101050A4939B2094939E221)
printk: 54 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8021
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160351319 ACK=598980542 WINDOW=42408
RES=0x00 ACK URGP=0 OPT (0101080A455889151B13F5C00101050A4942510149425659)
printk: 43 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8205
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160351559 ACK=599403254 WINDOW=49248
RES=0x00 ACK URGP=0 OPT (0101080A45588B011B13FA9B0101050A4948C4394948C991)
printk: 40 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8531
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160352039 ACK=600218582 WINDOW=45144
RES=0x00 ACK URGP=0 OPT (0101080A45588D371B1400160101050A4955351949553A71)
printk: 49 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8871
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160352519 ACK=601058534 WINDOW=38304
RES=0x00 ACK URGP=0 OPT (0101080A45588F521B1405500101050A4962062949620B81)
printk: 45 messages suppressed.
nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT=
SRC=100.100.100.100 DST=200.200.200.200 LEN=64 TOS=0x00 PREC=0x00 TTL=56
ID=8988
DF PROTO=TCP SPT=22 DPT=35858 SEQ=4160352663 ACK=601307510 WINDOW=41040
RES=0x00 ACK URGP=0 OPT (0101080A4558910A1B1409AB0101050A4965D2B94965D811)
Regards,
Sven
--
sven.riedel@securenet.de
SecureNet GmbH
Intranet & Internet Solutions
Frankfurter Ring 193a
D-80807 München
Tel: +49 89 32133-632
Fax: +49 89 32133-699
Zentrale: -600
www.securenet.de
Sitz der Gesellschaft: München
HRB München 118876
Geschäftsführer: Thomas Schreiber
next prev parent reply other threads:[~2008-03-26 10:21 UTC|newest]
Thread overview: 116+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-26 8:47 Transfer stalls with NAT under 2.6.24.3 Sven Riedel
2008-03-26 9:24 ` Patrick McHardy
2008-03-26 10:21 ` Sven Riedel [this message]
2008-03-26 15:47 ` Patrick McHardy
2008-03-26 18:45 ` Jozsef Kadlecsik
2008-03-26 19:16 ` Krzysztof Oledzki
2008-03-31 6:53 ` Sven Riedel
2008-07-04 14:54 ` TCP connection stalls under 2.6.24.7 Thomas Jarosch
2008-07-04 20:58 ` Jozsef Kadlecsik
2008-07-04 21:04 ` Jozsef Kadlecsik
2008-07-07 9:18 ` Thomas Jarosch
2008-07-07 13:18 ` Thomas Jarosch
2008-07-10 13:17 ` Jozsef Kadlecsik
2008-07-10 14:12 ` Thomas Jarosch
2008-07-10 21:21 ` Jozsef Kadlecsik
2008-07-11 14:33 ` Thomas Jarosch
2008-07-15 11:47 ` Thomas Jarosch
2008-07-15 16:10 ` Thomas Jarosch
2008-07-15 18:30 ` Dâniel Fraga
2008-07-31 4:47 ` Dâniel Fraga
2008-07-31 7:39 ` Ilpo Järvinen
2008-08-02 12:24 ` Dâniel Fraga
2008-07-15 20:17 ` Ilpo Järvinen
2008-07-16 8:07 ` Thomas Jarosch
2008-07-16 9:03 ` Thomas Jarosch
2008-07-17 13:55 ` Ilpo Järvinen
2008-07-17 15:15 ` Thomas Jarosch
2008-07-17 15:53 ` Ilpo Järvinen
2008-07-18 9:14 ` Thomas Jarosch
2008-07-18 13:55 ` Ilpo Järvinen
2008-07-18 14:02 ` Thomas Jarosch
2008-07-19 7:35 ` Ilpo Järvinen
2008-07-25 10:00 ` Ilpo Järvinen
2008-07-25 13:00 ` Thomas Jarosch
2008-07-25 14:06 ` Ilpo Järvinen
2008-07-25 15:34 ` Thomas Jarosch
2008-07-31 7:39 ` Thomas Jarosch
2008-07-31 12:44 ` Dâniel Fraga
2008-07-31 13:47 ` Thomas Jarosch
2008-07-31 14:11 ` Dâniel Fraga
2008-08-06 18:53 ` Dâniel Fraga
2008-08-07 6:54 ` Ilpo Järvinen
2008-08-07 11:50 ` Denys Fedoryshchenko
2008-08-07 12:11 ` Thomas Jarosch
2008-08-07 12:14 ` Ilpo Järvinen
2008-08-07 12:23 ` Denys Fedoryshchenko
2008-08-08 9:56 ` Ilpo Järvinen
2008-08-08 10:32 ` Denys Fedoryshchenko
2008-08-07 11:33 ` [PATCH] tcp FRTO: in-order-only "TCP proxy" fragility workaround Ilpo Järvinen
2008-08-08 4:42 ` Bill Fink
2008-08-08 10:32 ` Ilpo Järvinen
2008-08-11 21:44 ` David Miller
2008-08-12 7:46 ` Thomas Jarosch
2008-08-12 8:18 ` David Miller
2008-08-12 17:43 ` Dâniel Fraga
2008-08-12 17:52 ` Ilpo Järvinen
2008-08-13 17:53 ` Dâniel Fraga
2008-08-13 18:34 ` Ilpo Järvinen
2008-08-15 4:34 ` Dâniel Fraga
2008-08-15 7:06 ` Ilpo Järvinen
2008-08-15 21:35 ` Dâniel Fraga
2008-08-15 22:06 ` Ilpo Järvinen
2008-08-15 23:57 ` Dâniel Fraga
2008-08-16 2:15 ` Dâniel Fraga
2008-08-16 7:10 ` Ilpo Järvinen
2008-08-16 19:18 ` Ilpo Järvinen
2008-08-17 0:36 ` Dâniel Fraga
2008-08-19 10:38 ` Ilpo Järvinen
2008-08-20 0:34 ` Dâniel Fraga
2008-08-20 7:57 ` Ilpo Järvinen
2008-08-20 12:37 ` Ilpo Järvinen
2008-08-22 21:32 ` Dâniel Fraga
2008-08-22 21:37 ` David Miller
2008-08-23 14:14 ` Dâniel Fraga
2008-08-23 14:38 ` Ilpo Järvinen
2008-08-24 19:38 ` Dâniel Fraga
2008-08-26 14:10 ` Ilpo Järvinen
2008-08-26 14:32 ` Ilpo Järvinen
2008-08-26 17:18 ` Dâniel Fraga
2008-08-26 20:40 ` Ilpo Järvinen
2008-08-26 21:17 ` Dâniel Fraga
2008-08-27 10:22 ` Ilpo Järvinen
2008-08-27 19:51 ` Dâniel Fraga
2008-08-27 20:32 ` Ilpo Järvinen
2008-08-27 20:50 ` Dâniel Fraga
2008-08-27 21:25 ` Ilpo Järvinen
2008-08-27 21:42 ` Dâniel Fraga
2008-08-27 22:24 ` Dâniel Fraga
2008-08-28 21:49 ` Dâniel Fraga
2008-08-29 13:07 ` Ilpo Järvinen
2008-08-29 17:41 ` Dâniel Fraga
2008-09-01 7:11 ` Ilpo Järvinen
2008-08-30 6:56 ` Dâniel Fraga
2008-09-01 7:11 ` Ilpo Järvinen
2008-09-07 8:17 ` Dâniel Fraga
2008-09-08 10:27 ` Ilpo Järvinen
2008-09-08 20:20 ` Dâniel Fraga
2008-09-11 13:44 ` Ilpo Järvinen
2008-09-11 17:30 ` Dâniel Fraga
2008-09-12 10:16 ` Ilpo Järvinen
2008-09-13 23:31 ` Dâniel Fraga
2008-09-16 12:10 ` Ilpo Järvinen
2008-09-16 14:24 ` Dâniel Fraga
2008-09-17 10:23 ` Ilpo Järvinen
2008-09-18 20:35 ` Dâniel Fraga
2008-09-18 21:04 ` Ilpo Järvinen
2008-09-21 3:02 ` Dâniel Fraga
2008-09-22 4:23 ` Dâniel Fraga
2008-09-22 11:22 ` Ilpo Järvinen
2008-09-22 16:13 ` Dâniel Fraga
2008-09-15 19:42 ` Dâniel Fraga
2008-09-11 18:12 ` Dâniel Fraga
2008-08-15 21:59 ` Dâniel Fraga
2008-08-13 8:00 ` Thomas Jarosch
2008-08-22 21:18 ` Ilpo Järvinen
2008-08-11 21:41 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47EA2399.1080201@securenet.de \
--to=sr@securenet.de \
--cc=kaber@trash.net \
--cc=netfilter-devel@vger.kernel.org \
--cc=netfilter@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.