Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5434 bytes --]

Hello,

My colleague have just reported that he is unable to access "REGISTER 
HERE" page from the https://my.procurve.com/profile/index.aspx. Short 
debuging shows that his connection was dropped by the netfilter code 
running on fw/nat host:

02:06:29.573031 IP (tos 0x0, ttl 127, id 43914, offset 0, flags [DF], proto: TCP (6), length: 48) 195.177.210.11.6347 > 216.34.143.7.443: S, cksum 0xbb9c (correct), 3455050277:3455050277(0) win 65535 <mss 1460,nop,nop,sackOK>
02:06:29.756857 IP (tos 0x0, ttl 241, id 41244, offset 0, flags [DF], proto: TCP (6), length: 48) 216.34.143.7.443 > 195.177.210.11.6347: S, cksum 0x0655 (correct), 1733443080:1733443080(0) ack 3455050278 win 4140 <mss 1380,nop,nop,sackOK>
02:06:29.757021 IP (tos 0x0, ttl 127, id 43920, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.11.6347 > 216.34.143.7.443: ., cksum 0x42f5 (correct), ack 1733443081 win 65535
02:06:29.757871 IP (tos 0x0, ttl 127, id 43923, offset 0, flags [DF], proto: TCP (6), length: 142) 195.177.210.11.6347 > 216.34.143.7.443: P 3455050278:3455050380(102) ack 1733443081 win 65535
02:06:29.941610 IP (tos 0x0, ttl 241, id 41252, offset 0, flags [DF], proto: TCP (6), length: 162) 216.34.143.7.443 > 195.177.210.11.6347: P 1733443081:1733443203(122) ack 3455050380 win 4242
02:06:29.944767 IP (tos 0x0, ttl 127, id 43930, offset 0, flags [DF], proto: TCP (6), length: 83) 195.177.210.11.6347 > 216.34.143.7.443: P 3455050380:3455050423(43) ack 1733443203 win 65413
02:06:29.946115 IP (tos 0x0, ttl 127, id 43932, offset 0, flags [DF], proto: TCP (6), length: 1216) 195.177.210.11.6347 > 216.34.143.7.443: P 3455050423:3455051599(1176) ack 1733443203 win 65413
02:06:29.946221 IP (tos 0x0, ttl 127, id 43933, offset 0, flags [DF], proto: TCP (6), length: 1420) 195.177.210.11.6347 > 216.34.143.7.443: . 3455051599:3455052979(1380) ack 1733443203 win 65413
02:06:29.946241 IP (tos 0x0, ttl 127, id 43934, offset 0, flags [DF], proto: TCP (6), length: 689) 195.177.210.11.6347 > 216.34.143.7.443: P 3455052979:3455053628(649) ack 1733443203 win 65413
02:06:30.130106 IP (tos 0x0, ttl 241, id 41259, offset 0, flags [DF], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.11.6347: ., cksum 0x27fd (correct), ack 3455051599 win 5461
02:06:30.130125 IP (tos 0x0, ttl 241, id 41261, offset 0, flags [DF], proto: TCP (6), length: 52) 216.34.143.7.443 > 195.177.210.11.6347: ., cksum 0x6cd9 (correct), ack 3455051599 win 5461 <nop,nop,sack 1 {236696358:236697007}>
02:06:30.134620 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.11.6347 > 216.34.143.7.443: R, cksum 0xe333 (correct), 3455051599:3455051599(0) win 0
02:06:30.134629 IP (tos 0x0, ttl 241, id 41265, offset 0, flags [DF], proto: TCP (6), length: 52) 216.34.143.7.443 > 195.177.210.11.6347: ., cksum 0x6263 (correct), ack 3455053628 win 7490 <nop,nop,sack 1 {236694978:236697007}>
02:06:30.140908 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.11.6347 > 216.34.143.7.443: R, cksum 0xdb46 (correct), 3455053628:3455053628(0) win 0
02:06:30.140929 IP (tos 0x0, ttl 241, id 41269, offset 0, flags [DF], proto: TCP (6), length: 565) 216.34.143.7.443 > 195.177.210.11.6347: P 1733443203:1733443716(513) ack 3455053628 win 7490 <nop,nop,sack 1 {236694978:236697007}>
02:06:30.147216 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.11.6347 > 216.34.143.7.443: R, cksum 0xdb46 (correct), 3455053628:3455053628(0) win 0
02:06:32.523268 IP (tos 0x0, ttl 127, id 43966, offset 0, flags [DF], proto: TCP (6), length: 1420) 195.177.210.11.6347 > 216.34.143.7.443: . 3455051599:3455052979(1380) ack 1733443203 win 65413
02:06:32.707202 IP (tos 0x0, ttl 240, id 40250, offset 0, flags [none], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.11.6347: R, cksum 0x3dc8 (correct), 1733443203:1733443203(0) ack 3455051599 win 65413

Jul  1 02:06:30 fw1 kernel: nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT= SRC=216.34.143.7 DST=195.177.210.11 LEN=52 TOS=0x00 PREC=0x00 TTL=241 ID=41261 DF PROTO=TCP SPT=443 DPT=6347 SEQ=1733443203 ACK=3455051599 WINDOW=5461 RES=0x00 ACK URGP=0 OPT (0101050A0E1BB3260E1BB5AF)
Jul  1 02:06:30 fw1 kernel: nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT= SRC=216.34.143.7 DST=195.177.210.11 LEN=52 TOS=0x00 PREC=0x00 TTL=241 ID=41265 DF PROTO=TCP SPT=443 DPT=6347 SEQ=1733443203 ACK=3455053628 WINDOW=7490 RES=0x00 ACK URGP=0 OPT (0101050A0E1BADC20E1BB5AF)
Jul  1 02:06:30 fw1 kernel: nf_ct_tcp: ACK is over the upper bound (ACKed data not seen yet) IN= OUT= SRC=216.34.143.7 DST=195.177.210.11 LEN=565 TOS=0x00 PREC=0x00 TTL=241 ID=41269 DF PROTO=TCP SPT=443 DPT=6347 SEQ=1733443203 ACK=3455053628 WINDOW=7490 RES=0x00 ACK PSH URGP=0 OPT (0101050A0E1BADC20E1BB5AF)

AFAIK netfilter is supposed to drop such packet (without sending a RST), 
isn't it? So why the RST packet was sent?

Setting net.ipv4.netfilter.ip_conntrack_tcp_be_liberal solves the problem, 
but this is not a right fix and now the main question is: was this ACK 
really over the upper bound since when 
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled it is possible 
to access this page (of course with many netfilter warnings that "ACK is 
over the upper bound").

root@fw1:~# uname -r
2.6.20.11

Best regards,

 			Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1363 bytes --]



On Sun, 1 Jul 2007, Krzysztof Oledzki wrote:

> Hello,
>
> My colleague have just reported that he is unable to access "REGISTER HERE" 
> page from the https://my.procurve.com/profile/index.aspx. Short debuging 
> shows that his connection was dropped by the netfilter code running on fw/nat 
> host:
<CUT>
> AFAIK netfilter is supposed to drop such packet (without sending a RST), 
> isn't it? So why the RST packet was sent?

OK, this was easy. The RST was sent simply because the packed was not 
dropped but instead delivered to the local IP - there was no 
valid tuple to change (unnat) the packed destination. Setting:

iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP

make no more RSTs, only retransmisions from the 216.34.143.7. And yes, I 
have a patched kernel so I'm able to filter packets in a PREROUTING 
chain.

The rest of the question remains:

> Setting net.ipv4.netfilter.ip_conntrack_tcp_be_liberal solves the problem, 
> but this is not a right fix and now the main question is: was this ACK really 
> over the upper bound since when 
> net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled it is possible to 
> access this page (of course with many netfilter warnings that "ACK is over 
> the upper bound").
>
> root@fw1:~# uname -r
> 2.6.20.11


Best regards,

 				Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 171 bytes --]



> but instead delivered to the local IP - there was no valid tuple to change
I meant: "(...) to the local IP _stack_ (...)".

Best regards,

 			Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 6945 bytes --]



On Sun, 1 Jul 2007, Krzysztof Oledzki wrote:

>
>
> On Sun, 1 Jul 2007, Krzysztof Oledzki wrote:
>
>> Hello,
>> 
>> My colleague have just reported that he is unable to access "REGISTER HERE" 
>> page from the https://my.procurve.com/profile/index.aspx. Short debuging 
>> shows that his connection was dropped by the netfilter code running on 
>> fw/nat host:
> <CUT>
>> AFAIK netfilter is supposed to drop such packet (without sending a RST), 
>> isn't it? So why the RST packet was sent?
>
> OK, this was easy. The RST was sent simply because the packed was not dropped 
> but instead delivered to the local IP - there was no valid tuple to change 
> (unnat) the packed destination. Setting:
>
> iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP
>
> make no more RSTs, only retransmisions from the 216.34.143.7. And yes, I have 
> a patched kernel so I'm able to filter packets in a PREROUTING chain.
>
> The rest of the question remains:
>
>> Setting net.ipv4.netfilter.ip_conntrack_tcp_be_liberal solves the problem, 
>> but this is not a right fix and now the main question is: was this ACK 
>> really over the upper bound since when 
>> net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled it is possible to 
>> access this page (of course with many netfilter warnings that "ACK is over 
>> the upper bound").

Found this:

http://groups.google.pl/group/fa.openbsd.tech/browse_frm/thread/e27c7363b2c636b5/01ba6e0fa873cf42

Sounds familiar - it seems that there may be a crappy OpenBSD firewall 
lurking somewhere along the path. :(

What we can do with such packets? Maybe, when a ack is valid but a sack is 
not (as it is in this situation) we are able to remove such insane sack 
option(s) with a hope that this ACK itself may acknowledge something?

This is how this connection looks like when 
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled:

04:34:42.276125 IP (tos 0x0, ttl 127, id 10660, offset 0, flags [DF], proto: TCP (6), length: 48) 195.177.210.11.6747 > 216.34.143.7.443: S, cksum 0x4b25 (correct), 1809363748:1809363748(0) win 65535 <mss 1460,nop,nop,sackOK>
04:34:42.459601 IP (tos 0x0, ttl 241, id 65082, offset 0, flags [DF], proto: TCP (6), length: 48) 216.34.143.7.443 > 195.177.210.11.6747: S, cksum 0xa5a8 (correct), 3724064662:3724064662(0) ack 1809363749 win 4140 <mss 1380,nop,nop,sackOK>
04:34:42.459764 IP (tos 0x0, ttl 127, id 10665, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.11.6747 > 216.34.143.7.443: ., cksum 0xe248 (correct), ack 3724064663 win 65535
04:34:42.460215 IP (tos 0x0, ttl 127, id 10666, offset 0, flags [DF], proto: TCP (6), length: 142) 195.177.210.11.6747 > 216.34.143.7.443: P 1809363749:1809363851(102) ack 3724064663 win 65535
04:34:42.644004 IP (tos 0x0, ttl 241, id 65111, offset 0, flags [DF], proto: TCP (6), length: 162) 216.34.143.7.443 > 195.177.210.11.6747: P 3724064663:3724064785(122) ack 1809363851 win 4242
04:34:42.644766 IP (tos 0x0, ttl 127, id 10672, offset 0, flags [DF], proto: TCP (6), length: 83) 195.177.210.11.6747 > 216.34.143.7.443: P 1809363851:1809363894(43) ack 3724064785 win 65413
04:34:42.647212 IP (tos 0x0, ttl 127, id 10674, offset 0, flags [DF], proto: TCP (6), length: 1216) 195.177.210.11.6747 > 216.34.143.7.443: P 1809363894:1809365070(1176) ack 3724064785 win 65413
04:34:42.647362 IP (tos 0x0, ttl 127, id 10675, offset 0, flags [DF], proto: TCP (6), length: 1420) 195.177.210.11.6747 > 216.34.143.7.443: . 1809365070:1809366450(1380) ack 3724064785 win 65413
04:34:42.647381 IP (tos 0x0, ttl 127, id 10676, offset 0, flags [DF], proto: TCP (6), length: 689) 195.177.210.11.6747 > 216.34.143.7.443: P 1809366450:1809367099(649) ack 3724064785 win 65413
04:34:42.831152 IP (tos 0x0, ttl 241, id 65117, offset 0, flags [DF], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.11.6747: ., cksum 0xc750 (correct), ack 1809365070 win 5461
acked 1809365070, 1380 + 649 octets unacked yet (1809367099)

04:34:42.831173 IP (tos 0x0, ttl 241, id 65119, offset 0, flags [DF], proto: TCP (6), length: 52) 216.34.143.7.443 > 195.177.210.11.6747: ., cksum 0x7ee4 (correct), ack 1809365070 win 5461 <nop,nop,sack 1 {2713871907:2713872556}>
sacked last 649 octets

04:34:42.835709 IP (tos 0x0, ttl 241, id 65123, offset 0, flags [DF], proto: TCP (6), length: 52) 216.34.143.7.443 > 195.177.210.11.6747: ., cksum 0x746e (correct), ack 1809367099 win 7490 <nop,nop,sack 1 {2713870527:2713872556}>
sacked full 2029 octets + acked everyting to 1809367099

04:34:42.841979 IP (tos 0x0, ttl 241, id 65127, offset 0, flags [DF], proto: TCP (6), length: 565) 216.34.143.7.443 > 195.177.210.11.6747: P 3724064785:3724065298(513) ack 1809367099 win 7490 <nop,nop,sack 1 {2713870527:2713872556}>
(redundant) sacked full 2029 octets + acked everyting to 1809367099 + sending some data + PUSH

04:34:42.853077 IP (tos 0x0, ttl 127, id 10684, offset 0, flags [DF], proto: TCP (6), length: 1150) 195.177.210.11.6747 > 216.34.143.7.443: P 1809367099:1809368209(1110) ack 3724065298 win 64900

Bingo... :)

04:34:43.136310 IP (tos 0x0, ttl 241, id 65148, offset 0, flags [DF], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.11.6747: ., cksum 0xacc9 (correct), ack 1809368209 win 8600
04:34:43.383861 IP (tos 0x0, ttl 241, id 65264, offset 0, flags [DF], proto: TCP (6), length: 1420) 216.34.143.7.443 > 195.177.210.11.6747: . 3724065298:3724066678(1380) ack 1809368209 win 8600
04:34:43.386285 IP (tos 0x0, ttl 241, id 65265, offset 0, flags [DF], proto: TCP (6), length: 120) 216.34.143.7.443 > 195.177.210.11.6747: P 3724066678:3724066758(80) ack 1809368209 win 8600
04:34:43.390499 IP (tos 0x0, ttl 241, id 65267, offset 0, flags [DF], proto: TCP (6), length: 1420) 216.34.143.7.443 > 195.177.210.11.6747: . 3724066758:3724068138(1380) ack 1809368209 win 8600
04:34:43.394683 IP (tos 0x0, ttl 241, id 65270, offset 0, flags [DF], proto: TCP (6), length: 1420) 216.34.143.7.443 > 195.177.210.11.6747: . 3724068138:3724069518(1380) ack 1809368209 win 8600
04:34:43.403021 IP (tos 0x0, ttl 127, id 10695, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.11.6747 > 216.34.143.7.443: ., cksum 0xc8ad (correct), ack 3724066758 win 65535
(...)

There is a Windows XP host behind this NAT and is seems it is quite happy 
ignoring this sack crap as the ack itself finnaly acknowledged everyting. 
The transmition continues.


Additionally, creating TCPOPTSSTRIP target to allow striping specific tcp 
option(s) (for example Sack-Permitted from a SYN packet) may also be 
usable if it is possible to include this extension in a base kernel. This 
may also help with a similar window scaling problem as current solution 
requires to add a route on _all_ hosts inside a network. Working around it 
on a firewall may be much faster.


Best regards,

 				Krzysztof Olędzki

PS: I love to talk with myself. ;)

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Jan Engelhardt]

On Jul 1 2007 05:01, Krzysztof Oledzki wrote:
>>
>> > Setting net.ipv4.netfilter.ip_conntrack_tcp_be_liberal solves the problem,
>> > but this is not a right fix and now the main question is: was this ACK
>> > really over the upper bound since when
>> > net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled it is possible to
>> > access this page (of course with many netfilter warnings that "ACK is over
>> > the upper bound").
>
> Found this:
>
> http://groups.google.pl/group/fa.openbsd.tech/browse_frm/thread/e27c7363b2c636b5/01ba6e0fa873cf42
>
> Sounds familiar - it seems that there may be a crappy OpenBSD firewall lurking
> somewhere along the path. :(

Question... does the problem also go away if you leave
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal as previously and instead set
net.ipv4.tcp_sack = 0?


	Jan
-- 

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4298 bytes --]



On Sun, 1 Jul 2007, Jan Engelhardt wrote:

>
> On Jul 1 2007 05:01, Krzysztof Oledzki wrote:
>>>
>>>> Setting net.ipv4.netfilter.ip_conntrack_tcp_be_liberal solves the problem,
>>>> but this is not a right fix and now the main question is: was this ACK
>>>> really over the upper bound since when
>>>> net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled it is possible to
>>>> access this page (of course with many netfilter warnings that "ACK is over
>>>> the upper bound").
>>
>> Found this:
>>
>> http://groups.google.pl/group/fa.openbsd.tech/browse_frm/thread/e27c7363b2c636b5/01ba6e0fa873cf42
>>
>> Sounds familiar - it seems that there may be a crappy OpenBSD firewall lurking
>> somewhere along the path. :(
>
> Question... does the problem also go away if you leave
> net.ipv4.netfilter.ip_conntrack_tcp_be_liberal as previously and instead set
> net.ipv4.tcp_sack = 0?

On the firewall/nat itself? No. But disabling sack on that workstation by 
setting SackOpts=0 (DWORD) in the 
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" 
indeed makes this problem go away as there was no sack negotiated:

16:52:43.238991 IP (tos 0x0, ttl 127, id 5645, offset 0, flags [DF], proto: TCP (6), length: 44) 195.177.210.97.1058 > 216.34.143.7.443: S, cksum 0x5dc1 (correct), 4217846755:4217846755(0) win 65535 <mss 1460>
16:52:43.422683 IP (tos 0x0, ttl 241, id 20704, offset 0, flags [DF], proto: TCP (6), length: 44) 216.34.143.7.443 > 195.177.210.97.1058: S, cksum 0xac20 (correct), 3439121590:3439121590(0) ack 4217846756 win 4140 <mss 1380>
No sackOK in this SYN.

16:52:43.422836 IP (tos 0x0, ttl 127, id 5650, offset 0, flags [DF], proto: TCP (6), length: 40) 195.177.210.97.1058 > 216.34.143.7.443: ., cksum 0xd3b9 (correct), ack 3439121591 win 65535
16:52:43.423199 IP (tos 0x0, ttl 127, id 5651, offset 0, flags [DF], proto: TCP (6), length: 142) 195.177.210.97.1058 > 216.34.143.7.443: P, cksum 0xfcd6 (correct), 4217846756:4217846858(102) ack 3439121591 win 65535
16:52:43.607074 IP (tos 0x0, ttl 241, id 20741, offset 0, flags [DF], proto: TCP (6), length: 162) 216.34.143.7.443 > 195.177.210.97.1058: P, cksum 0xd004 (correct), 3439121591:3439121713(122) ack 4217846858 win 4242
16:52:43.607737 IP (tos 0x0, ttl 127, id 5662, offset 0, flags [DF], proto: TCP (6), length: 83) 195.177.210.97.1058 > 216.34.143.7.443: P, cksum 0x672d (correct), 4217846858:4217846901(43) ack 3439121713 win 65413
16:52:43.608728 IP (tos 0x0, ttl 127, id 5663, offset 0, flags [DF], proto: TCP (6), length: 1139) 195.177.210.97.1058 > 216.34.143.7.443: P, cksum 0xad36 (correct), 4217846901:4217848000(1099) ack 3439121713 win 65413
16:52:43.608842 IP (tos 0x0, ttl 127, id 5664, offset 0, flags [DF], proto: TCP (6), length: 1420) 195.177.210.97.1058 > 216.34.143.7.443: ., cksum 0x00c6 (correct), 4217848000:4217849380(1380) ack 3439121713 win 65413
16:52:43.608963 IP (tos 0x0, ttl 127, id 5665, offset 0, flags [DF], proto: TCP (6), length: 689) 195.177.210.97.1058 > 216.34.143.7.443: P, cksum 0xcaad (correct), 4217849380:4217850029(649) ack 3439121713 win 65413
Last octet is 4217850029...

16:52:43.792713 IP (tos 0x0, ttl 241, id 20753, offset 0, flags [DF], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.97.1058: ., cksum 0xb95b (correct), ack 4217848000 win 5384
16:52:43.792839 IP (tos 0x0, ttl 241, id 20755, offset 0, flags [DF], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.97.1058: ., cksum 0xb95b (correct), ack 4217848000 win 5384
Two redundant acks for 4217848000... (?)

16:52:43.792965 IP (tos 0x0, ttl 241, id 20759, offset 0, flags [DF], proto: TCP (6), length: 40) 216.34.143.7.443 > 195.177.210.97.1058: ., cksum 0xa981 (correct), ack 4217850029 win 7413
Finally acking 4217850029

16:52:43.798334 IP (tos 0x0, ttl 241, id 20763, offset 0, flags [DF], proto: TCP (6), length: 553) 216.34.143.7.443 > 195.177.210.97.1058: P, cksum 0x9a72 (correct), 3439121713:3439122226(513) ack 4217850029 win 7413
16:52:43.799601 IP (tos 0x0, ttl 127, id 5674, offset 0, flags [DF], proto: TCP (6), length: 1073) 195.177.210.97.1058 > 216.34.143.7.443: P, cksum 0xd49e (correct), 4217850029:4217851062(1033) ack 3439122226 win 64900

(...) :)

Best regards,


 				Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Jan Engelhardt]

On Jul 1 2007 17:03, Krzysztof Oledzki wrote:
>
> On the firewall/nat itself? No. But disabling sack on that workstation by

On the workstation, yes.
It was a windows one? Oops, I missed that.


	Jan
-- 

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Krzysztof Oledzki wrote:
> Found this:
> 
> http://groups.google.pl/group/fa.openbsd.tech/browse_frm/thread/e27c7363b2c636b5/01ba6e0fa873cf42
> 
> 
> Sounds familiar - it seems that there may be a crappy OpenBSD firewall
> lurking somewhere along the path. :(


Indeed, too bad they apparently don't fix their crap and we're getting
at least one report per month about this.


> What we can do with such packets? Maybe, when a ack is valid but a sack
> is not (as it is in this situation) we are able to remove such insane
> sack option(s) with a hope that this ACK itself may acknowledge something?


I'm not too big a fan of this idea, but I will consider it if someone
sends a patch. It would have to be manually enabled at least.

> This is how this connection looks like when
> net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is enabled:
> 
> [...]
> 04:34:42.835709 IP (tos 0x0, ttl 241, id 65123, offset 0, flags [DF],
> proto: TCP (6), length: 52) 216.34.143.7.443 > 195.177.210.11.6747: .,
> cksum 0x746e (correct), ack 1809367099 win 7490 <nop,nop,sack 1
> {2713870527:2713872556}>
> sacked full 2029 octets + acked everyting to 1809367099
> 
> 04:34:42.841979 IP (tos 0x0, ttl 241, id 65127, offset 0, flags [DF],
> proto: TCP (6), length: 565) 216.34.143.7.443 > 195.177.210.11.6747: P
> 3724064785:3724065298(513) ack 1809367099 win 7490 <nop,nop,sack 1
> {2713870527:2713872556}>
> (redundant) sacked full 2029 octets + acked everyting to 1809367099 +
> sending some data + PUSH
> 
> 04:34:42.853077 IP (tos 0x0, ttl 127, id 10684, offset 0, flags [DF],
> proto: TCP (6), length: 1150) 195.177.210.11.6747 > 216.34.143.7.443: P
> 1809367099:1809368209(1110) ack 3724065298 win 64900
> 
> Bingo... :)
> 
> There is a Windows XP host behind this NAT and is seems it is quite
> happy ignoring this sack crap as the ack itself finnaly acknowledged
> everyting. The transmition continues.
> 
> 
> Additionally, creating TCPOPTSSTRIP target to allow striping specific
> tcp option(s) (for example Sack-Permitted from a SYN packet) may also be
> usable if it is possible to include this extension in a base kernel.
> This may also help with a similar window scaling problem as current
> solution requires to add a route on _all_ hosts inside a network.
> Working around it on a firewall may be much faster.


Feel free to send patches :)

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1924 bytes --]



On Mon, 2 Jul 2007, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>> Found this:
>>
>> http://groups.google.pl/group/fa.openbsd.tech/browse_frm/thread/e27c7363b2c636b5/01ba6e0fa873cf42
>>
>>
>> Sounds familiar - it seems that there may be a crappy OpenBSD firewall
>> lurking somewhere along the path. :(
>
>
> Indeed, too bad they apparently don't fix their crap and we're getting
> at least one report per month about this.

It seems they finally fixed it in a cvs at end of the Jan 2006 (so late - 
10 years after sack had been specified in rfc2018): 
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r1=1.508&r2=1.509

AFAIK it first went into 4.0 (released Nov 1, 2006) and also OPENBSD_3_9 
(STABLE "branch" for 3.9) so it is safe to assume that only very new 
installations may be safe. :(

AFAIK (again) this fix hasn't went into FreeBSD and NetBSD at all. :( Oh, 
crappy...

>> What we can do with such packets? Maybe, when a ack is valid but a sack
>> is not (as it is in this situation) we are able to remove such insane
>> sack option(s) with a hope that this ACK itself may acknowledge something?
>
>
> I'm not too big a fan of this idea, but I will consider it if someone
> sends a patch. It would have to be manually enabled at least.

Fair enough.

>> Additionally, creating TCPOPTSSTRIP target to allow striping specific
>> tcp option(s) (for example Sack-Permitted from a SYN packet) may also be
>> usable if it is possible to include this extension in a base kernel.
>> This may also help with a similar window scaling problem as current
>> solution requires to add a route on _all_ hosts inside a network.
>> Working around it on a firewall may be much faster.
>
>
> Feel free to send patches :)

OK. Will try to cook something. Can I base it on the IPV4OPTSSTRIP 
there is a better example? :)

Best regards,

 				Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Krzysztof Oledzki wrote:
> On Mon, 2 Jul 2007, Patrick McHardy wrote:
>> Krzysztof Oledzki wrote:
>>
>>> Sounds familiar - it seems that there may be a crappy OpenBSD firewall
>>> lurking somewhere along the path. :(
>>
>>
>>
>> Indeed, too bad they apparently don't fix their crap and we're getting
>> at least one report per month about this.
> 
> 
> It seems they finally fixed it in a cvs at end of the Jan 2006 (so late
> - 10 years after sack had been specified in rfc2018):
> http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r1=1.508&r2=1.509
> 
> 
> AFAIK it first went into 4.0 (released Nov 1, 2006) and also OPENBSD_3_9
> (STABLE "branch" for 3.9) so it is safe to assume that only very new
> installations may be safe. :(
> 
> AFAIK (again) this fix hasn't went into FreeBSD and NetBSD at all. :(
> Oh, crappy...


Thanks for the information.

>>> Additionally, creating TCPOPTSSTRIP target to allow striping specific
>>> tcp option(s) (for example Sack-Permitted from a SYN packet) may also be
>>> usable if it is possible to include this extension in a base kernel.
>>> This may also help with a similar window scaling problem as current
>>> solution requires to add a route on _all_ hosts inside a network.
>>> Working around it on a firewall may be much faster.
>>
>>
>>
>> Feel free to send patches :)
> 
> 
> OK. Will try to cook something. Can I base it on the IPV4OPTSSTRIP there
> is a better example? :)


Its not a great example IIRC, but I don't know of a better one.

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Krzysztof Oledzki wrote:
> 
> OK, this was easy. The RST was sent simply because the packed was not
> dropped but instead delivered to the local IP - there was no valid tuple
> to change (unnat) the packed destination. Setting:
> 
> iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP


We should really document that with window tracking and NAT you
must drop INVALID packets to avoid them getting delivered locally
and causing a RST.

> 
> make no more RSTs, only retransmisions from the 216.34.143.7. And yes, I
> have a patched kernel so I'm able to filter packets in a PREROUTING chain.

Dropping works without any patches.

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Jozsef Kadlecsik]

On Mon, 2 Jul 2007, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>>
>> OK, this was easy. The RST was sent simply because the packed was not
>> dropped but instead delivered to the local IP - there was no valid tuple
>> to change (unnat) the packed destination. Setting:
>>
>> iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP
>
> We should really document that with window tracking and NAT you
> must drop INVALID packets to avoid them getting delivered locally
> and causing a RST.

Couldn't we do it in NAT itself? I.e drop the packet by NAT if 
ip_conntrack_tcp_be_liberal is unset and the packet is INVALID.

It'd be backward-incompatible as it'd change the current behaviour but 
were more safer (saner) than the present approach.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
           H-1525 Budapest 114, POB. 49, Hungary

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Jozsef Kadlecsik wrote:
> On Mon, 2 Jul 2007, Patrick McHardy wrote:
> 
>> Krzysztof Oledzki wrote:
>>
>>>
>>> OK, this was easy. The RST was sent simply because the packed was not
>>> dropped but instead delivered to the local IP - there was no valid tuple
>>> to change (unnat) the packed destination. Setting:
>>>
>>> iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP
>>
>>
>> We should really document that with window tracking and NAT you
>> must drop INVALID packets to avoid them getting delivered locally
>> and causing a RST.
> 
> 
> Couldn't we do it in NAT itself? I.e drop the packet by NAT if
> ip_conntrack_tcp_be_liberal is unset and the packet is INVALID.


I'm a bit sceptical about NAT core caring about TCP conntrack
specific sysctls, I'd prefer an unconditional drop.

> It'd be backward-incompatible as it'd change the current behaviour but
> were more safer (saner) than the present approach.


Yes, for this case it definitely makes sense. I'm just wondering
whether we'll break anything. Tools like nmap come to mind ..

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Jozsef Kadlecsik]

On Mon, 2 Jul 2007, Patrick McHardy wrote:

>>> We should really document that with window tracking and NAT you
>>> must drop INVALID packets to avoid them getting delivered locally
>>> and causing a RST.
>>
>> Couldn't we do it in NAT itself? I.e drop the packet by NAT if
>> ip_conntrack_tcp_be_liberal is unset and the packet is INVALID.
>
> I'm a bit sceptical about NAT core caring about TCP conntrack
> specific sysctls, I'd prefer an unconditional drop.

NAT works on top of conntrack so peeking the flags were not completely 
perverse ;-). If NAT drops INVALID packets undconditionally, that'd 
disable the sysctl flag completely and we had to say "this sysctl setting 
cannot be used if the NAT module is loaded in".

>> It'd be backward-incompatible as it'd change the current behaviour but
>> were more safer (saner) than the present approach.
>
> Yes, for this case it definitely makes sense. I'm just wondering
> whether we'll break anything. Tools like nmap come to mind ..

Dropping packets is a good thing. :-)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
           H-1525 Budapest 114, POB. 49, Hungary

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Jozsef Kadlecsik wrote:
> On Mon, 2 Jul 2007, Patrick McHardy wrote:
> 
>>>> We should really document that with window tracking and NAT you
>>>> must drop INVALID packets to avoid them getting delivered locally
>>>> and causing a RST.
>>>
>>>
>>> Couldn't we do it in NAT itself? I.e drop the packet by NAT if
>>> ip_conntrack_tcp_be_liberal is unset and the packet is INVALID.
>>
>>
>> I'm a bit sceptical about NAT core caring about TCP conntrack
>> specific sysctls, I'd prefer an unconditional drop.
> 
> 
> NAT works on top of conntrack so peeking the flags were not completely
> perverse ;-). If NAT drops INVALID packets undconditionally, that'd
> disable the sysctl flag completely and we had to say "this sysctl
> setting cannot be used if the NAT module is loaded in".


Would it really? The sysctls flag makes *more* packets be regarded
as valid, so I'm not I'm following ..


> 
>>> It'd be backward-incompatible as it'd change the current behaviour but
>>> were more safer (saner) than the present approach.
>>
>>
>> Yes, for this case it definitely makes sense. I'm just wondering
>> whether we'll break anything. Tools like nmap come to mind ..
> 
> 
> Dropping packets is a good thing. :-)


Incoming yes, outgoing .. who knows :)

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Jozsef Kadlecsik]

On Mon, 2 Jul 2007, Patrick McHardy wrote:

>>> I'm a bit sceptical about NAT core caring about TCP conntrack
>>> specific sysctls, I'd prefer an unconditional drop.
>>
>> NAT works on top of conntrack so peeking the flags were not completely
>> perverse ;-). If NAT drops INVALID packets undconditionally, that'd
>> disable the sysctl flag completely and we had to say "this sysctl
>> setting cannot be used if the NAT module is loaded in".
>
> Would it really? The sysctls flag makes *more* packets be regarded
> as valid, so I'm not I'm following ..

You are right, sigh, I wrote rubbish.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
           H-1525 Budapest 114, POB. 49, Hungary

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1442 bytes --]



On Mon, 2 Jul 2007, Patrick McHardy wrote:

> Jozsef Kadlecsik wrote:
>> On Mon, 2 Jul 2007, Patrick McHardy wrote:
>>
>>> Krzysztof Oledzki wrote:
>>>
>>>>
>>>> OK, this was easy. The RST was sent simply because the packed was not
>>>> dropped but instead delivered to the local IP - there was no valid tuple
>>>> to change (unnat) the packed destination. Setting:
>>>>
>>>> iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP
>>>
>>>
>>> We should really document that with window tracking and NAT you
>>> must drop INVALID packets to avoid them getting delivered locally
>>> and causing a RST.
>>
>>
>> Couldn't we do it in NAT itself? I.e drop the packet by NAT if
>> ip_conntrack_tcp_be_liberal is unset and the packet is INVALID.
>
>
> I'm a bit sceptical about NAT core caring about TCP conntrack
> specific sysctls, I'd prefer an unconditional drop.
>
>> It'd be backward-incompatible as it'd change the current behaviour but
>> were more safer (saner) than the present approach.
>
>
> Yes, for this case it definitely makes sense. I'm just wondering
> whether we'll break anything. Tools like nmap come to mind ..

I think it may be better to drop it with iptables. Before drop it is 
possible to log (-j LOG/ULOG/...) such packets - this is what I'm doing 
for example. It is often worth to know about dropped packets with INVALID 
state.

Best regards,

 				Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Krzysztof Oledzki wrote:
> I think it may be better to drop it with iptables. Before drop it is
> possible to log (-j LOG/ULOG/...) such packets - this is what I'm doing
> for example. It is often worth to know about dropped packets with
> INVALID state.


Good point.

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1103 bytes --]



On Mon, 2 Jul 2007, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>>
>> OK, this was easy. The RST was sent simply because the packed was not
>> dropped but instead delivered to the local IP - there was no valid tuple
>> to change (unnat) the packed destination. Setting:
>>
>> iptables -I PREROUTING -m conntrack --ctstate INVALID -j DROP
>
>
> We should really document that with window tracking and NAT you
> must drop INVALID packets to avoid them getting delivered locally
> and causing a RST.

Indeed. There should be a big, fat warning about dropping in INPUT (and 
probably FORWARD). The question is where: Kconfig (NAT)? man iptables? 
both? ;)

>> make no more RSTs, only retransmisions from the 216.34.143.7. And yes, I
>> have a patched kernel so I'm able to filter packets in a PREROUTING chain.
>
> Dropping works without any patches.

Yes, in INPUT. I discovered that such packets goes to INPUT shortly after 
I had written this mail. Before that I had put this in PREROUTING, which 
is not possible by default.

Best regards,

 			Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Krzysztof Oledzki wrote:
> On Mon, 2 Jul 2007, Patrick McHardy wrote:
> 
>> We should really document that with window tracking and NAT you
>> must drop INVALID packets to avoid them getting delivered locally
>> and causing a RST.
> 
> 
> Indeed. There should be a big, fat warning about dropping in INPUT (and
> probably FORWARD). The question is where: Kconfig (NAT)? man iptables?
> both? ;)


The manpage I guess. Kconfig is not really the place for this IMO.

>>> make no more RSTs, only retransmisions from the 216.34.143.7. And yes, I
>>> have a patched kernel so I'm able to filter packets in a PREROUTING
>>> chain.
>>
>>
>> Dropping works without any patches.
> 
> 
> Yes, in INPUT. I discovered that such packets goes to INPUT shortly
> after I had written this mail. Before that I had put this in PREROUTING,
> which is not possible by default.


You can drop in PREROUTING/mangle for example. In the filter table
its not possible of course since there is no PREROUTING chain :)

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1310 bytes --]



On Mon, 2 Jul 2007, Patrick McHardy wrote:

> Krzysztof Oledzki wrote:
>> On Mon, 2 Jul 2007, Patrick McHardy wrote:
>>
>>> We should really document that with window tracking and NAT you
>>> must drop INVALID packets to avoid them getting delivered locally
>>> and causing a RST.
>>
>>
>> Indeed. There should be a big, fat warning about dropping in INPUT (and
>> probably FORWARD). The question is where: Kconfig (NAT)? man iptables?
>> both? ;)
>
>
> The manpage I guess. Kconfig is not really the place for this IMO.
OK.

>>>> make no more RSTs, only retransmisions from the 216.34.143.7. And yes, I
>>>> have a patched kernel so I'm able to filter packets in a PREROUTING
>>>> chain.
>>>
>>>
>>> Dropping works without any patches.
>>
>>
>> Yes, in INPUT. I discovered that such packets goes to INPUT shortly
>> after I had written this mail. Before that I had put this in PREROUTING,
>> which is not possible by default.
>
>
> You can drop in PREROUTING/mangle for example. In the filter table
> its not possible of course since there is no PREROUTING chain :)

Indeed, but I found dropping in filter more useful. That's why I mention 
about my patched kernel. BTW: is there any reason why it is not available 
by default?

Best regards,

 			Krzysztof Olędzki

Re: Problem accessing https://my.procurve.com/profile/index.aspx (ACK is over the upper bound)

[Patrick McHardy]

Krzysztof Oledzki wrote:
>>
>> You can drop in PREROUTING/mangle for example. In the filter table
>> its not possible of course since there is no PREROUTING chain :)
> 
> 
> Indeed, but I found dropping in filter more useful. That's why I mention
> about my patched kernel. BTW: is there any reason why it is not
> available by default?


You mean PREROUTING/filter? Mainly performance, new hooks are expensive.