* PATCH: "invalid SYNIN=" - a patch and a question @ 2007-09-26 12:27 Krzysztof Oledzki 2007-10-03 13:06 ` Krzysztof Oledzki 2007-10-04 5:45 ` Patrick McHardy 0 siblings, 2 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-09-26 12:27 UTC (permalink / raw) To: Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 1349 bytes --] Hello, Attached patch should fix missing space between "SYN" and "IN=". nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.16 DST=192.168.50.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=19810 DF PROTO=TCP SPT=43183 DPT=80 SEQ=3917241971 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A3C14363B0000000001030307) UID=451 192.168.150.16 <- my local ip address 192.168.50.21 <- remote server My question is about this message. Apparently there is someting wrong with my configuration. I'm running http proxy/load balancer so my server makes _a lot_ of connections for/to the same address IP pair. I noticed that many of them were not successful. I know about TIMEWAIT issue, but as long there are enough free ports for current connection rate (in my situation it is about 100/s) it should not be a problem. So, with net.netfilter.nf_conntrack_log_invalid set to 255 I get: grep "invalid SYNIN" /var/log/syslog |wc -l 1186 Could the problem be that default net.netfilter.nf_conntrack_tcp_timeout_time_wait is 120s by default, and TCP_TIMEWAIT_LEN is 60: /usr/src/linux/include/net/tcp.h:#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT I tried to change net.netfilter.nf_conntrack_tcp_timeout_time_wait to 60 but this does not help much. Best regards, Krzysztof Olędzki [-- Attachment #2: Type: TEXT/PLAIN, Size: 778 bytes --] [NETFILTER]: Trivial fix for "nf_ct_tcp: invalid SYN" This nf_log_packet needs a space char. Trivial fix Signed-off-by: Krzysztof Piotr Oledzki <ole@ans.pl> diff -Nur linux-2.6.23-rc8-orig/net/netfilter/nf_conntrack_proto_tcp.c linux-2.6.23-rc8/net/netfilter/nf_conntrack_proto_tcp.c --- linux-2.6.23-rc8-orig/net/netfilter/nf_conntrack_proto_tcp.c 2007-09-25 02:33:10.000000000 +0200 +++ linux-2.6.23-rc8/net/netfilter/nf_conntrack_proto_tcp.c 2007-09-25 16:50:39.000000000 +0200 @@ -897,7 +897,7 @@ write_unlock_bh(&tcp_lock); if (LOG_INVALID(IPPROTO_TCP)) nf_log_packet(pf, 0, skb, NULL, NULL, - NULL, "nf_ct_tcp: invalid SYN"); + NULL, "nf_ct_tcp: invalid SYN "); return -NF_ACCEPT; } case TCP_CONNTRACK_CLOSE: ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-09-26 12:27 PATCH: "invalid SYNIN=" - a patch and a question Krzysztof Oledzki @ 2007-10-03 13:06 ` Krzysztof Oledzki 2007-10-03 13:54 ` Jozsef Kadlecsik ` (2 more replies) 2007-10-04 5:45 ` Patrick McHardy 1 sibling, 3 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-03 13:06 UTC (permalink / raw) To: Netfilter Developer Mailing List; +Cc: kaber [-- Attachment #1: Type: TEXT/PLAIN, Size: 4597 bytes --] On Wed, 26 Sep 2007, Krzysztof Oledzki wrote: > Hello, > > Attached patch should fix missing space between "SYN" and "IN=". > > nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.16 DST=192.168.50.21 LEN=60 > TOS=0x00 PREC=0x00 TTL=64 ID=19810 DF PROTO=TCP SPT=43183 DPT=80 > SEQ=3917241971 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT > (020405B40402080A3C14363B0000000001030307) UID=451 > > 192.168.150.16 <- my local ip address > 192.168.50.21 <- remote server > > My question is about this message. Apparently there is someting wrong with my > configuration. I'm running http proxy/load balancer so my server makes _a > lot_ of connections for/to the same address IP pair. I noticed that many of > them were not successful. I know about TIMEWAIT issue, but as long there are > enough free ports for current connection rate (in my situation it is about > 100/s) it should not be a problem. > > So, with net.netfilter.nf_conntrack_log_invalid set to 255 I get: > > grep "invalid SYNIN" /var/log/syslog |wc -l > 1186 > > Could the problem be that default > net.netfilter.nf_conntrack_tcp_timeout_time_wait is 120s by default, and > TCP_TIMEWAIT_LEN is 60: > > /usr/src/linux/include/net/tcp.h:#define TCP_TIMEWAIT_LEN (60*HZ) /* how long > to wait to destroy TIME-WAIT > > I tried to change net.netfilter.nf_conntrack_tcp_timeout_time_wait to 60 but > this does not help much. Small update to this issue. It seems that both ip stack and netfilter indeed handle connections using different timers: # wget --bind-address 192.168.0.1 192.168.129.28 -O /dev/null --14:36:00-- http://192.168.129.28/ => `/dev/null' Connecting to 192.168.129.28:80... connected. HTTP request sent, awaiting response... 200 OK Length: 915 [text/html] 100%[==========================================================================================>] 915 --.--K/s 14:36:00 (13.43 MB/s) - `/dev/null' saved [915/915] # conntrack -L |grep =192.168.129.28|grep 192.168.0.1: tcp 6 119 TIME_WAIT src=192.168.0.1 dst=192.168.129.28 sport=18300 dport=80 packets=6 bytes=422 src=192.168.129.28 dst=192.168.0.1 sport=80 dport=18300 packets=4 bytes=1475 [ASSURED] mark=0 use=1 # ss -anto |egrep "192.168.0.1.*192.168.129.28": TIME-WAIT 0 0 192.168.0.1:18300 192.168.129.28:80 timer:(timewait,58sec,0) After 60s kernel is able to reuse this (18300) port but it will be blocked by local netfilter with this "invalid SYN" message. This is especially more likely in newer kernels with tcp port randomization. This does not solve my problem but it think we should consider changing net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait from 120s to 60s. I also wondering if the code from nf_conntrack_proto_tcp.c is correct: --- cut here --- new_state = tcp_conntracks[dir][index][old_state]; switch (new_state) (...) case TCP_CONNTRACK_SYN_SENT: if (old_state < TCP_CONNTRACK_TIME_WAIT) break; if ((conntrack->proto.tcp.seen[dir].flags & IP_CT_TCP_FLAG_CLOSE_INIT) || after(ntohl(th->seq), conntrack->proto.tcp.seen[dir].td_end)) { /* Attempt to reopen a closed connection. * Delete this connection and look up again. */ write_unlock_bh(&tcp_lock); if (del_timer(&conntrack->timeout)) conntrack->timeout.function((unsigned long) conntrack); return -NF_REPEAT; } else { write_unlock_bh(&tcp_lock); if (LOG_INVALID(IPPROTO_TCP)) nf_log_packet(pf, 0, skb, NULL, NULL, NULL, "nf_ct_tcp: invalid SYN"); return -NF_ACCEPT; } --- cut here --- It seems that tcp_conntracks allows such (TCP_CONNTRACK_TIME_WAIT -> TCP_CONNTRACK_SYN_SENT) transition, pointing to rfc1122: When a connection is closed actively, it MUST linger in TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). However, it MAY accept a new SYN from the remote TCP to reopen the connection directly from TIME-WAIT state, if it: (1) assigns its initial sequence number for the new connection to be larger than the largest sequence number it used on the previous connection incarnation, and (2) returns to TIME-WAIT state if the SYN turns out to be an old duplicate. So, it seems that this "after(...)" does not match this packet, right? Patric, what do you think about this? Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-03 13:06 ` Krzysztof Oledzki @ 2007-10-03 13:54 ` Jozsef Kadlecsik 2007-10-03 19:57 ` Krzysztof Oledzki 2007-10-04 5:59 ` Patrick McHardy 2007-10-04 6:15 ` Patrick McHardy 2007-10-10 19:26 ` Jozsef Kadlecsik 2 siblings, 2 replies; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-03 13:54 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Patrick McHardy Hi, On Wed, 3 Oct 2007, Krzysztof Oledzki wrote: > I also wondering if the code from nf_conntrack_proto_tcp.c is correct: [...] I think the code segment is correct. However, the end of the segment is miscalculated by segment_seq_plus_len because the offset added to 'dataoff' by the get_l4proto function of IPv4/IPv6 is not taken into account. Please check whether the patch below fixes it: Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> diff -urN --exclude-from=/usr/src/diff.exclude linux-2.6.23-rc4-orig/net/netfilter/nf_conntrack_proto_tcp.c linux-2.6.23-rc4-tcp_win/net/netfilter/nf_conntrack_proto_tcp.c --- linux-2.6.23-rc4-orig/net/netfilter/nf_conntrack_proto_tcp.c 2007-08-28 03:32:35.000000000 +0200 +++ linux-2.6.23-rc4-tcp_win/net/netfilter/nf_conntrack_proto_tcp.c 2007-10-03 15:44:01.000000000 +0200 @@ -509,7 +509,8 @@ seq = ntohl(tcph->seq); ack = sack = ntohl(tcph->ack_seq); win = ntohs(tcph->window); - end = segment_seq_plus_len(seq, skb->len, dataoff, tcph); + end = segment_seq_plus_len(seq, skb->len, + dataoff - skb_network_offset(skb), tcph); if (receiver->flags & IP_CT_TCP_FLAG_SACK_PERM) tcp_sack(skb, dataoff, tcph, &sack); Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-03 13:54 ` Jozsef Kadlecsik @ 2007-10-03 19:57 ` Krzysztof Oledzki 2007-10-04 7:26 ` Jozsef Kadlecsik 2007-10-04 5:59 ` Patrick McHardy 1 sibling, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-03 19:57 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Netfilter Developer Mailing List, Patrick McHardy [-- Attachment #1: Type: TEXT/PLAIN, Size: 995 bytes --] On Wed, 3 Oct 2007, Jozsef Kadlecsik wrote: > Hi, > > On Wed, 3 Oct 2007, Krzysztof Oledzki wrote: > >> I also wondering if the code from nf_conntrack_proto_tcp.c is correct: > [...] > > I think the code segment is correct. However, the end of the segment > is miscalculated by segment_seq_plus_len because the offset added to > 'dataoff' by the get_l4proto function of IPv4/IPv6 is not taken into > account. Please check whether the patch below fixes it: Thank you. Currently I am traveling so I do not have direct access to this host but I will test it ASAP. > Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> > > diff -urN --exclude-from=/usr/src/diff.exclude linux-2.6.23-rc4-orig/net/netfilter/nf_conntrack_proto_tcp.c linux-2.6.23-rc4-tcp_win/net/netfilter/nf_conntrack_proto_tcp.c Does this same error exist in 2.6.22? BTW: what is your opinion about net.netfilter.nf_conntrack_tcp_timeout_time_wait? Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-03 19:57 ` Krzysztof Oledzki @ 2007-10-04 7:26 ` Jozsef Kadlecsik 0 siblings, 0 replies; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-04 7:26 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Patrick McHardy On Wed, 3 Oct 2007, Krzysztof Oledzki wrote: > Does this same error exist in 2.6.22? Yes, and in earlier releases as well: as far as I see the bug was introduced by nf_conntrack. > BTW: what is your opinion about > net.netfilter.nf_conntrack_tcp_timeout_time_wait? We already "violate" the RFC. And considering other IP stacks, in order to avoid all unnecessarily blocked late packets, we should probably set the timeout value back to 2*MSL. By setting it to 1MSL we try to find the proper balance between "not too much blocked late packets" and "get rid of conntrack entries as fast as possible". Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-03 13:54 ` Jozsef Kadlecsik 2007-10-03 19:57 ` Krzysztof Oledzki @ 2007-10-04 5:59 ` Patrick McHardy 2007-10-04 7:52 ` Jozsef Kadlecsik 1 sibling, 1 reply; 40+ messages in thread From: Patrick McHardy @ 2007-10-04 5:59 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List Jozsef Kadlecsik wrote: > I think the code segment is correct. However, the end of the segment > is miscalculated by segment_seq_plus_len because the offset added to > 'dataoff' by the get_l4proto function of IPv4/IPv6 is not taken into > account. Please check whether the patch below fixes it: Actually that offset should always be zero for anything but packets encapsulated within ICMP, which don't pass though TCP conntrack. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-04 5:59 ` Patrick McHardy @ 2007-10-04 7:52 ` Jozsef Kadlecsik 2007-10-04 7:59 ` Patrick McHardy 0 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-04 7:52 UTC (permalink / raw) To: Patrick McHardy; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List Hi Patrick, On Thu, 4 Oct 2007, Patrick McHardy wrote: > Jozsef Kadlecsik wrote: > > I think the code segment is correct. However, the end of the segment > > is miscalculated by segment_seq_plus_len because the offset added to > > 'dataoff' by the get_l4proto function of IPv4/IPv6 is not taken into > > account. Please check whether the patch below fixes it: > > Actually that offset should always be zero for anything but packets > encapsulated within ICMP, which don't pass though TCP conntrack. Here we calculate the sequence number of the last octet in the packet from the actual data length of the skbuff, so by the 'dataoff' argument of segment_seq_plus_len we must pass the length of the IP(v4|6) header. But the 'dataoff' parameter supplied by the nf_conntrack core is larger by skb_network_offset(skb) therefore 'dataoff' must be decremented by it before passing to segment_seq_plus_len. Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-04 7:52 ` Jozsef Kadlecsik @ 2007-10-04 7:59 ` Patrick McHardy 2007-10-04 8:41 ` Jozsef Kadlecsik 0 siblings, 1 reply; 40+ messages in thread From: Patrick McHardy @ 2007-10-04 7:59 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List Jozsef Kadlecsik wrote: > Hi Patrick, > > On Thu, 4 Oct 2007, Patrick McHardy wrote: > >>Jozsef Kadlecsik wrote: >> >>>I think the code segment is correct. However, the end of the segment >>>is miscalculated by segment_seq_plus_len because the offset added to >>>'dataoff' by the get_l4proto function of IPv4/IPv6 is not taken into >>>account. Please check whether the patch below fixes it: >> >>Actually that offset should always be zero for anything but packets >>encapsulated within ICMP, which don't pass though TCP conntrack. > > > Here we calculate the sequence number of the last octet in the packet from > the actual data length of the skbuff, so by the 'dataoff' argument of > segment_seq_plus_len we must pass the length of the IP(v4|6) header. But > the 'dataoff' parameter supplied by the nf_conntrack core is larger by > skb_network_offset(skb) therefore 'dataoff' must be decremented by it > before passing to segment_seq_plus_len. Right, I didn't express myself correctly. What I meant to say is that skb_network_offset is always 0 while we're inside the IP(v6) layer. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-04 7:59 ` Patrick McHardy @ 2007-10-04 8:41 ` Jozsef Kadlecsik 2007-10-05 20:06 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-04 8:41 UTC (permalink / raw) To: Patrick McHardy; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List On Thu, 4 Oct 2007, Patrick McHardy wrote: > Right, I didn't express myself correctly. What I meant to say is that > skb_network_offset is always 0 while we're inside the IP(v6) layer. Then what I sent is a NOOP, I'm sorry for the noise. Krzysztof, could you send us tcpdump recording as Patrick suggested? Adding '#define DEBUG' to the front of nf_conntrack_proto_tcp.c could help a lot too, as we could see what conntrack thinks about the packets. Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-04 8:41 ` Jozsef Kadlecsik @ 2007-10-05 20:06 ` Krzysztof Oledzki 2007-10-08 9:32 ` Jozsef Kadlecsik 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-05 20:06 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 8832 bytes --] On Thu, 4 Oct 2007, Jozsef Kadlecsik wrote: > On Thu, 4 Oct 2007, Patrick McHardy wrote: > >> Right, I didn't express myself correctly. What I meant to say is that >> skb_network_offset is always 0 while we're inside the IP(v6) layer. > > Then what I sent is a NOOP, I'm sorry for the noise. > > Krzysztof, could you send us tcpdump recording as Patrick suggested? Si. --- example #1 begin --- Connections from 192.168.150.12:38485 21:44:59.822546 IP (tos 0x0, ttl 1, id 42330, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.12.38485 > 192.168.50.21.80: S, cksum 0x4a98 (correct), 3235584816:3235584816(0) win 5840 <mss 1460,sackOK,timestamp 1812577898 0,nop,wscale 7> 21:44:59.822643 IP (tos 0x0, ttl 128, id 13168, offset 0, flags [none], proto TCP (6), length 64) 192.168.50.21.80 > 192.168.150.12.38485: S, cksum 0x272e (correct), 2584049816:2584049816(0) ack 3235584817 win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> 21:44:59.822677 IP (tos 0x0, ttl 1, id 42331, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.12.38485 > 192.168.50.21.80: ., cksum 0x7d57 (correct), ack 2584049817 win 46 <nop,nop,timestamp 1812577898 0> 21:44:59.822703 IP (tos 0x0, ttl 1, id 42332, offset 0, flags [DF], proto TCP (6), length 935) 192.168.150.12.38485 > 192.168.50.21.80: P 3235584817:3235585700(883) ack 2584049817 win 46 <nop,nop,timestamp 1812577898 0> 21:44:59.822992 IP (tos 0x0, ttl 128, id 13169, offset 0, flags [DF], proto TCP (6), length 308) 192.168.50.21.80 > 192.168.150.12.38485: FP 2584049817:2584050073(256) ack 3235585700 win 64652 <nop,nop,timestamp 32205200 1812577898> 21:44:59.862142 IP (tos 0x0, ttl 1, id 42333, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.12.38485 > 192.168.50.21.80: ., cksum 0x0d38 (correct), ack 2584050074 win 54 <nop,nop,timestamp 1812577938 32205200> 21:44:59.870913 IP (tos 0x0, ttl 1, id 42334, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.12.38485 > 192.168.50.21.80: F, cksum 0x0d2f (correct), 3235585700:3235585700(0) ack 2584050074 win 54 <nop,nop,timestamp 1812577946 32205200> 21:44:59.870998 IP (tos 0x0, ttl 128, id 13216, offset 0, flags [DF], proto TCP (6), length 52) 192.168.50.21.80 > 192.168.150.12.38485: ., cksum 0x10d7 (correct), ack 3235585701 win 64652 <nop,nop,timestamp 32205201 1812577946> 21:45:03.379006 IP (tos 0x0, ttl 1, id 63436, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.12.38485 > 192.168.50.21.80: S, cksum 0x4896 (correct), 2494249856:2494249856(0) win 5840 <mss 1460,sackOK,timestamp 1812581452 0,nop,wscale 7> 21:45:06.380974 IP (tos 0x0, ttl 1, id 63437, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.12.38485 > 192.168.50.21.80: S, cksum 0x3cdc (correct), 2494249856:2494249856(0) win 5840 <mss 1460,sackOK,timestamp 1812584454 0,nop,wscale 7> Oct 5 21:45:03 fw1 kernel: nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.12 DST=192.168.50.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63436 DF PROTO=TCP SPT=38485 DPT=80 SEQ=2494249856 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A6C09CC4C0000000001030307) UID=451 Oct 5 21:45:06 fw1 kernel: nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.12 DST=192.168.50.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63437 DF PROTO=TCP SPT=38485 DPT=80 SEQ=2494249856 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A6C09D8060000000001030307) UID=451 --- example #1 end --- --- example #1 begin --- Connections from 192.168.150.13:60522 21:39:44.134429 IP (tos 0x0, ttl 1, id 49813, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.13.60522 > 192.168.50.51.80: S, cksum 0x3234 (correct), 1066479575:1066479575(0) win 5840 <mss 1460,sackOK,timestamp 1812262210 0,nop,wscale 7> 21:39:44.134526 IP (tos 0x0, ttl 128, id 29436, offset 0, flags [none], proto TCP (6), length 64) 192.168.50.51.80 > 192.168.150.13.60522: S, cksum 0xda9c (correct), 2447238080:2447238080(0) ack 1066479576 win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> 21:39:44.134559 IP (tos 0x0, ttl 1, id 49814, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.51.80: ., cksum 0x01f3 (correct), ack 2447238081 win 46 <nop,nop,timestamp 1812262210 0> 21:39:44.134585 IP (tos 0x0, ttl 1, id 49815, offset 0, flags [DF], proto TCP (6), length 475) 192.168.150.13.60522 > 192.168.50.51.80: P 1066479576:1066479999(423) ack 2447238081 win 46 <nop,nop,timestamp 1812262210 0> 21:39:44.134826 IP (tos 0x0, ttl 128, id 29437, offset 0, flags [DF], proto TCP (6), length 164) 192.168.50.51.80 > 192.168.150.13.60522: FP 2447238081:2447238193(112) ack 1066479999 win 65112 <nop,nop,timestamp 12462265 1812262210> 21:39:44.152045 IP (tos 0x0, ttl 1, id 49816, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.51.80: F, cksum 0xd651 (correct), 1066479999:1066479999(0) ack 2447238194 win 46 <nop,nop,timestamp 1812262227 12462265> 21:39:44.152247 IP (tos 0x0, ttl 128, id 29444, offset 0, flags [DF], proto TCP (6), length 52) 192.168.50.51.80 > 192.168.150.13.60522: ., cksum 0xd826 (correct), ack 1066480000 win 65112 <nop,nop,timestamp 12462265 1812262227> 21:44:12.884700 IP (tos 0x0, ttl 1, id 34935, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.13.60522 > 192.168.50.61.80: S, cksum 0x258d (correct), 3536555879:3536555879(0) win 5840 <mss 1460,sackOK,timestamp 1812530960 0,nop,wscale 7> 21:44:12.884856 IP (tos 0x0, ttl 128, id 7132, offset 0, flags [none], proto TCP (6), length 64) 192.168.50.61.80 > 192.168.150.13.60522: S, cksum 0x93b0 (correct), 2820415897:2820415897(0) ack 3536555880 win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> 21:44:12.884889 IP (tos 0x0, ttl 1, id 34936, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.61.80: ., cksum 0xa134 (correct), ack 2820415898 win 46 <nop,nop,timestamp 1812530960 0> 21:44:12.884914 IP (tos 0x0, ttl 1, id 34937, offset 0, flags [DF], proto TCP (6), length 354) 192.168.150.13.60522 > 192.168.50.61.80: P 3536555880:3536556182(302) ack 2820415898 win 46 <nop,nop,timestamp 1812530960 0> 21:44:12.885855 IP (tos 0x0, ttl 128, id 32768, offset 0, flags [none], proto TCP (6), length 430) 192.168.50.61.80 > 192.168.150.13.60522: P 2820415898:2820416276(378) ack 3536556182 win 65535 <nop,nop,timestamp 32197611 1812530960> 21:44:12.885880 IP (tos 0x0, ttl 1, id 34938, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.61.80: ., cksum 0x50ad (correct), ack 2820416276 win 54 <nop,nop,timestamp 1812530961 32197611> 21:44:12.885888 IP (tos 0x0, ttl 128, id 32769, offset 0, flags [none], proto TCP (6), length 52) 192.168.50.61.80 > 192.168.150.13.60522: F, cksum 0x50e3 (correct), 2820416276:2820416276(0) ack 3536556182 win 65535 <nop,nop,timestamp 32197611 1812530960> 21:44:12.925141 IP (tos 0x0, ttl 1, id 34939, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.61.80: ., cksum 0x5084 (correct), ack 2820416277 win 54 <nop,nop,timestamp 1812531001 32197611> 21:44:12.943021 IP (tos 0x0, ttl 1, id 34940, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.61.80: F, cksum 0x5072 (correct), 3536556182:3536556182(0) ack 2820416277 win 54 <nop,nop,timestamp 1812531018 32197611> 21:44:12.943142 IP (tos 0x0, ttl 128, id 32770, offset 0, flags [none], proto TCP (6), length 52) 192.168.50.61.80 > 192.168.150.13.60522: ., cksum 0x5037 (correct), ack 3536556183 win 65534 <nop,nop,timestamp 32197725 1812531018> 21:44:21.462090 IP (tos 0x0, ttl 1, id 60902, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.13.60522 > 192.168.50.61.80: S, cksum 0xcef7 (correct), 3521103209:3521103209(0) win 5840 <mss 1460,sackOK,timestamp 1812539535 0,nop,wscale 7> 21:44:24.461155 IP (tos 0x0, ttl 1, id 60903, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.13.60522 > 192.168.50.61.80: S, cksum 0xc33d (correct), 3521103209:3521103209(0) win 5840 <mss 1460,sackOK,timestamp 1812542537 0,nop,wscale 7> Oct 5 21:44:21 fw1 kernel: nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.13 DST=192.168.50.61 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=60902 DF PROTO=TCP SPT=60522 DPT=80 SEQ=3521103209 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A6C09288F0000000001030307) UID=451 --- example #2 end--- Please let me know if you need more tcpdumps, etc. If that may help I can also provide "conntrack -E" output. > Adding '#define DEBUG' to the front of nf_conntrack_proto_tcp.c could help > a lot too, as we could see what conntrack thinks about the packets. This has to wait one week - I do not want to do it remotely. I will do it ASAP when I come back from Greece. Thank you. Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-05 20:06 ` Krzysztof Oledzki @ 2007-10-08 9:32 ` Jozsef Kadlecsik 2007-10-08 9:45 ` Jozsef Kadlecsik ` (2 more replies) 0 siblings, 3 replies; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-08 9:32 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Patrick McHardy, Netfilter Developer Mailing List Hi Krzysztof, On Fri, 5 Oct 2007, Krzysztof Oledzki wrote: > --- example #1 begin --- [...] > 21:44:59.870913 IP (tos 0x0, ttl 1, id 42334, offset 0, flags [DF], proto TCP > (6), length 52) 192.168.150.12.38485 > 192.168.50.21.80: F, cksum 0x0d2f > (correct), 3235585700:3235585700(0) ack 2584050074 win 54 <nop,nop,timestamp > 1812577946 32205200> > 21:44:59.870998 IP (tos 0x0, ttl 128, id 13216, offset 0, flags [DF], proto > TCP (6), length 52) 192.168.50.21.80 > 192.168.150.12.38485: ., cksum 0x10d7 > (correct), ack 3235585701 win 64652 <nop,nop,timestamp 32205201 1812577946> > > 21:45:03.379006 IP (tos 0x0, ttl 1, id 63436, offset 0, flags [DF], proto TCP > (6), length 60) 192.168.150.12.38485 > 192.168.50.21.80: S, cksum 0x4896 > (correct), 2494249856:2494249856(0) win 5840 <mss 1460,sackOK,timestamp > 1812581452 0,nop,wscale 7> > > Oct 5 21:45:03 fw1 kernel: nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.12 > DST=192.168.50.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63436 DF PROTO=TCP > SPT=38485 DPT=80 SEQ=2494249856 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT > (020405B40402080A6C09CC4C0000000001030307) UID=451 The last sequence number ACK-ed by the server is 3235585701. The ISN sent by the client at reopening is 2494249856, which is not after the largest sequence number used in the previous session. > --- example #1 begin --- [...] > 21:44:12.943021 IP (tos 0x0, ttl 1, id 34940, offset 0, flags [DF], proto TCP > (6), length 52) 192.168.150.13.60522 > 192.168.50.61.80: F, cksum 0x5072 > (correct), 3536556182:3536556182(0) ack 2820416277 win 54 <nop,nop,timestamp > 1812531018 32197611> > 21:44:12.943142 IP (tos 0x0, ttl 128, id 32770, offset 0, flags [none], proto > TCP (6), length 52) 192.168.50.61.80 > 192.168.150.13.60522: ., cksum 0x5037 > (correct), ack 3536556183 win 65534 <nop,nop,timestamp 32197725 1812531018> > > 21:44:21.462090 IP (tos 0x0, ttl 1, id 60902, offset 0, flags [DF], proto TCP > (6), length 60) 192.168.150.13.60522 > 192.168.50.61.80: S, cksum 0xcef7 > (correct), 3521103209:3521103209(0) win 5840 <mss 1460,sackOK,timestamp > 1812539535 0,nop,wscale 7> And the same here: largest seq is 3536556183, but the ISN is 3521103209. It seems to me conntack is just right. Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 9:32 ` Jozsef Kadlecsik @ 2007-10-08 9:45 ` Jozsef Kadlecsik 2007-10-08 13:56 ` Krzysztof Oledzki 2007-10-08 13:58 ` Krzysztof Oledzki 2007-10-08 16:39 ` Patrick McHardy 2 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-08 9:45 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Patrick McHardy, Netfilter Developer Mailing List On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: [...] > It seems to me conntack is just right. Isn't there an ISN randomizator/anonymizer gear between the client and the server? Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 9:45 ` Jozsef Kadlecsik @ 2007-10-08 13:56 ` Krzysztof Oledzki 2007-10-08 14:01 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-08 13:56 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 430 bytes --] On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: > On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: > > [...] >> It seems to me conntack is just right. > > Isn't there an ISN randomizator/anonymizer gear between the client and the > server? No. It is a direct connection between the LB (Linux 2.6.22) and Windows servers. As you may noticed there is even a GID number in my log. Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 13:56 ` Krzysztof Oledzki @ 2007-10-08 14:01 ` Krzysztof Oledzki 0 siblings, 0 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-08 14:01 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 517 bytes --] On Mon, 8 Oct 2007, Krzysztof Oledzki wrote: > > > On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: > >> On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: >> >> [...] >>> It seems to me conntack is just right. >> >> Isn't there an ISN randomizator/anonymizer gear between the client and the >> server? > > No. It is a direct connection between the LB (Linux 2.6.22) and Windows > servers. As you may noticed there is even a GID number in my log. s/GID/UID/ ;) Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 9:32 ` Jozsef Kadlecsik 2007-10-08 9:45 ` Jozsef Kadlecsik @ 2007-10-08 13:58 ` Krzysztof Oledzki 2007-10-08 14:01 ` Jozsef Kadlecsik 2007-10-08 16:39 ` Patrick McHardy 2 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-08 13:58 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 2581 bytes --] On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: > Hi Krzysztof, > > On Fri, 5 Oct 2007, Krzysztof Oledzki wrote: > >> --- example #1 begin --- > [...] >> 21:44:59.870913 IP (tos 0x0, ttl 1, id 42334, offset 0, flags [DF], proto TCP >> (6), length 52) 192.168.150.12.38485 > 192.168.50.21.80: F, cksum 0x0d2f >> (correct), 3235585700:3235585700(0) ack 2584050074 win 54 <nop,nop,timestamp >> 1812577946 32205200> >> 21:44:59.870998 IP (tos 0x0, ttl 128, id 13216, offset 0, flags [DF], proto >> TCP (6), length 52) 192.168.50.21.80 > 192.168.150.12.38485: ., cksum 0x10d7 >> (correct), ack 3235585701 win 64652 <nop,nop,timestamp 32205201 1812577946> >> >> 21:45:03.379006 IP (tos 0x0, ttl 1, id 63436, offset 0, flags [DF], proto TCP >> (6), length 60) 192.168.150.12.38485 > 192.168.50.21.80: S, cksum 0x4896 >> (correct), 2494249856:2494249856(0) win 5840 <mss 1460,sackOK,timestamp >> 1812581452 0,nop,wscale 7> >> >> Oct 5 21:45:03 fw1 kernel: nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.150.12 >> DST=192.168.50.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=63436 DF PROTO=TCP >> SPT=38485 DPT=80 SEQ=2494249856 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT >> (020405B40402080A6C09CC4C0000000001030307) UID=451 > > The last sequence number ACK-ed by the server is 3235585701. The ISN sent > by the client at reopening is 2494249856, which is not after the largest > sequence number used in the previous session. > >> --- example #1 begin --- > [...] >> 21:44:12.943021 IP (tos 0x0, ttl 1, id 34940, offset 0, flags [DF], proto TCP >> (6), length 52) 192.168.150.13.60522 > 192.168.50.61.80: F, cksum 0x5072 >> (correct), 3536556182:3536556182(0) ack 2820416277 win 54 <nop,nop,timestamp >> 1812531018 32197611> >> 21:44:12.943142 IP (tos 0x0, ttl 128, id 32770, offset 0, flags [none], proto >> TCP (6), length 52) 192.168.50.61.80 > 192.168.150.13.60522: ., cksum 0x5037 >> (correct), ack 3536556183 win 65534 <nop,nop,timestamp 32197725 1812531018> >> >> 21:44:21.462090 IP (tos 0x0, ttl 1, id 60902, offset 0, flags [DF], proto TCP >> (6), length 60) 192.168.150.13.60522 > 192.168.50.61.80: S, cksum 0xcef7 >> (correct), 3521103209:3521103209(0) win 5840 <mss 1460,sackOK,timestamp >> 1812539535 0,nop,wscale 7> > > And the same here: largest seq is 3536556183, but the ISN is 3521103209. > > It seems to me conntack is just right. So? This bug is in Linux IP stack, as it should not reopen this port or reopen using a different SEQ, right? It seems I should go to a different ML. :| Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 13:58 ` Krzysztof Oledzki @ 2007-10-08 14:01 ` Jozsef Kadlecsik 2007-10-08 17:28 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-08 14:01 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Patrick McHardy, Netfilter Developer Mailing List On Mon, 8 Oct 2007, Krzysztof Oledzki wrote: > > It seems to me conntack is just right. > > So? This bug is in Linux IP stack, as it should not reopen this port or reopen > using a different SEQ, right? It seems I should go to a different ML. :| What is the exact kernel version of this machine? Do you have got any kernel patches which might touch the IP stack installed? Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 14:01 ` Jozsef Kadlecsik @ 2007-10-08 17:28 ` Krzysztof Oledzki 2007-10-09 8:11 ` Jozsef Kadlecsik 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-08 17:28 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 710 bytes --] On Mon, 8 Oct 2007, Jozsef Kadlecsik wrote: > On Mon, 8 Oct 2007, Krzysztof Oledzki wrote: > >>> It seems to me conntack is just right. >> >> So? This bug is in Linux IP stack, as it should not reopen this port or reopen >> using a different SEQ, right? It seems I should go to a different ML. :| > > What is the exact kernel version of this machine? It is 2.6.22.6 exactly. AFAIK there is a new "tcp source port randomization" code included in 2.6.21 or 2.6.22. Maybe it is somehow related? Or maybe it only make the problem easier to trigger? > Do you have got any kernel patches which might touch the IP stack > installed? No, I hope. ;) Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 17:28 ` Krzysztof Oledzki @ 2007-10-09 8:11 ` Jozsef Kadlecsik 2007-10-09 13:17 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-09 8:11 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Patrick McHardy, Netfilter Developer Mailing List On Mon, 8 Oct 2007, Krzysztof Oledzki wrote: > > What is the exact kernel version of this machine? > It is 2.6.22.6 exactly. AFAIK there is a new "tcp source port randomization" > code included in 2.6.21 or 2.6.22. Maybe it is somehow related? Or maybe it > only make the problem easier to trigger? I have downloaded and installed 2.6.22.6 and couldn't reproduce it. Have you got any non-default networking setting (like tcp_tw_reuse enabled)? Or, maybe, some special networking kernel option (TCP MD5 sigs, for example)? Could you please instruct me what to do exactly to reproduce the problem? Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-09 8:11 ` Jozsef Kadlecsik @ 2007-10-09 13:17 ` Krzysztof Oledzki 2007-10-09 13:22 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-09 13:17 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 1366 bytes --] On Tue, 9 Oct 2007, Jozsef Kadlecsik wrote: > On Mon, 8 Oct 2007, Krzysztof Oledzki wrote: > >>> What is the exact kernel version of this machine? >> It is 2.6.22.6 exactly. AFAIK there is a new "tcp source port randomization" >> code included in 2.6.21 or 2.6.22. Maybe it is somehow related? Or maybe it >> only make the problem easier to trigger? > > I have downloaded and installed 2.6.22.6 and couldn't reproduce it. Have > you got any non-default networking setting (like tcp_tw_reuse enabled)? No: net.ipv4.tcp_tw_reuse = 0 > Or, maybe, some special networking kernel option (TCP MD5 sigs, for > example)? No: zcat /proc/config.gz |grep MD5S # CONFIG_TCP_MD5SIG is not set > Could you please instruct me what to do exactly to reproduce > the problem? --- cut here --- sysctl net.ipv4.ip_local_port_range="50000 50003" sysctl net.netfilter.nf_conntrack_log_invalid=255 while true ; do echo -ne "HEAD / HTTP/1.0\r\nHost: www.wp.pl\r\n\r\n"|nc wp.pl 80 ; sleep 1 ; done --- cut here --- It takes a few seconds to generate a lot of "SEQ is under the lower bound" and "ACK is under the lower bound" and finally "invalid SYNIN=". In my original configuration ip_local_port_range is much, much bigger but there are >> 100 new connections per second initiated. Thank you. Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-09 13:17 ` Krzysztof Oledzki @ 2007-10-09 13:22 ` Krzysztof Oledzki 0 siblings, 0 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-09 13:22 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Patrick McHardy, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 625 bytes --] On Tue, 9 Oct 2007, Krzysztof Oledzki wrote: > > > On Tue, 9 Oct 2007, Jozsef Kadlecsik wrote: > >> On Mon, 8 Oct 2007, Krzysztof Oledzki wrote: >> >>>> What is the exact kernel version of this machine? >>> It is 2.6.22.6 exactly. AFAIK there is a new "tcp source port >>> randomization" >>> code included in 2.6.21 or 2.6.22. Maybe it is somehow related? Or maybe >>> it >>> only make the problem easier to trigger? >> >> I have downloaded and installed 2.6.22.6 and couldn't reproduce it. BTW: this problem also exists with 2.6.22.10-rc1, just checked. Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 9:32 ` Jozsef Kadlecsik 2007-10-08 9:45 ` Jozsef Kadlecsik 2007-10-08 13:58 ` Krzysztof Oledzki @ 2007-10-08 16:39 ` Patrick McHardy 2007-10-08 17:23 ` Krzysztof Oledzki 2 siblings, 1 reply; 40+ messages in thread From: Patrick McHardy @ 2007-10-08 16:39 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List Jozsef Kadlecsik wrote: > Hi Krzysztof, > >>--- example #1 begin --- > > [...] > > The last sequence number ACK-ed by the server is 3235585701. The ISN sent > by the client at reopening is 2494249856, which is not after the largest > sequence number used in the previous session. > > >>--- example #1 begin --- > > [...] > > > And the same here: largest seq is 3536556183, but the ISN is 3521103209. > > It seems to me conntack is just right. thats true, but I'm wondering, is there any benefit in being strict about this? The chances of accidentally reopening an old connection are a lot smaller than breaking things as in this case. Or maybe we could add PAWS checks, although that would increase the conntrack size by another 8 bytes. Krzysztof, does the problem disappear if you use something like 30 s for the TIME_WAIT timeout? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-08 16:39 ` Patrick McHardy @ 2007-10-08 17:23 ` Krzysztof Oledzki 0 siblings, 0 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-08 17:23 UTC (permalink / raw) To: Patrick McHardy; +Cc: Jozsef Kadlecsik, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 2567 bytes --] On Mon, 8 Oct 2007, Patrick McHardy wrote: > Jozsef Kadlecsik wrote: >> Hi Krzysztof, >> >>> --- example #1 begin --- >> >> [...] >> >> The last sequence number ACK-ed by the server is 3235585701. The ISN sent >> by the client at reopening is 2494249856, which is not after the largest >> sequence number used in the previous session. >> >> >>> --- example #1 begin --- >> >> [...] >> >> >> And the same here: largest seq is 3536556183, but the ISN is 3521103209. >> >> It seems to me conntack is just right. > > > thats true, but I'm wondering, is there any benefit in being > strict about this? The chances of accidentally reopening an > old connection are a lot smaller than breaking things as in > this case. Or maybe we could add PAWS checks, although that > would increase the conntrack size by another 8 bytes. > > Krzysztof, does the problem disappear if you use something > like 30 s for the TIME_WAIT timeout? No, as a delay between a new and an old happens to be <30s, for example: * ~4s: 21:44:59.870913 IP (tos 0x0, ttl 1, id 42334, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.12.38485 > 192.168.50.21.80: F, cksum 0x0d2f (correct), 3235585700:3235585700(0) ack 2584050074 win 54 <nop,nop,timestamp 1812577946 32205200> 21:44:59.870998 IP (tos 0x0, ttl 128, id 13216, offset 0, flags [DF], proto TCP (6), length 52) 192.168.50.21.80 > 192.168.150.12.38485: ., cksum 0x10d7 (correct), ack 3235585701 win 64652 <nop,nop,timestamp 32205201 1812577946> 21:45:03.379006 IP (tos 0x0, ttl 1, id 63436, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.12.38485 > 192.168.50.21.80: S, cksum 0x4896 (correct), 2494249856:2494249856(0) win 5840 <mss 1460,sackOK,timestamp 1812581452 0,nop,wscale 7> * ~28s: 21:39:44.152045 IP (tos 0x0, ttl 1, id 49816, offset 0, flags [DF], proto TCP (6), length 52) 192.168.150.13.60522 > 192.168.50.51.80: F, cksum 0xd651 (correct), 1066479999:1066479999(0) ack 2447238194 win 46 <nop,nop,timestamp 1812262227 12462265> 21:39:44.152247 IP (tos 0x0, ttl 128, id 29444, offset 0, flags [DF], proto TCP (6), length 52) 192.168.50.51.80 > 192.168.150.13.60522: ., cksum 0xd826 (correct), ack 1066480000 win 65112 <nop,nop,timestamp 12462265 1812262227> 21:44:12.884700 IP (tos 0x0, ttl 1, id 34935, offset 0, flags [DF], proto TCP (6), length 60) 192.168.150.13.60522 > 192.168.50.61.80: S, cksum 0x258d (correct), 3536555879:3536555879(0) win 5840 <mss 1460,sackOK,timestamp 1812530960 0,nop,wscale 7> Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-03 13:06 ` Krzysztof Oledzki 2007-10-03 13:54 ` Jozsef Kadlecsik @ 2007-10-04 6:15 ` Patrick McHardy 2007-10-10 19:26 ` Jozsef Kadlecsik 2 siblings, 0 replies; 40+ messages in thread From: Patrick McHardy @ 2007-10-04 6:15 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Jozsef Kadlecsik Krzysztof Oledzki wrote: > Small update to this issue. It seems that both ip stack and netfilter > indeed handle connections using different timers: > > # wget --bind-address 192.168.0.1 192.168.129.28 -O /dev/null > > --14:36:00-- http://192.168.129.28/ > => `/dev/null' > Connecting to 192.168.129.28:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 915 [text/html] > > 100%[==========================================================================================>] > 915 --.--K/s > > 14:36:00 (13.43 MB/s) - `/dev/null' saved [915/915] > > # conntrack -L |grep =192.168.129.28|grep 192.168.0.1: > tcp 6 119 TIME_WAIT src=192.168.0.1 dst=192.168.129.28 sport=18300 > dport=80 packets=6 bytes=422 src=192.168.129.28 dst=192.168.0.1 sport=80 > dport=18300 packets=4 bytes=1475 [ASSURED] mark=0 use=1 > > # ss -anto |egrep "192.168.0.1.*192.168.129.28": > TIME-WAIT 0 0 192.168.0.1:18300 > 192.168.129.28:80 timer:(timewait,58sec,0) > > > After 60s kernel is able to reuse this (18300) port but it will be > blocked by local netfilter with this "invalid SYN" message. This is > especially more likely in newer kernels with tcp port randomization. > > This does not solve my problem but it think we should consider changing > net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait from 120s to 60s. As you quoted below, we should use 2*MSL, which would actually be 240s. Not sure why Linux TCP uses 60s .. > > I also wondering if the code from nf_conntrack_proto_tcp.c is correct: > > --- cut here --- > new_state = tcp_conntracks[dir][index][old_state]; > > switch (new_state) > (...) > case TCP_CONNTRACK_SYN_SENT: > if (old_state < TCP_CONNTRACK_TIME_WAIT) > break; > if ((conntrack->proto.tcp.seen[dir].flags & > IP_CT_TCP_FLAG_CLOSE_INIT) > || after(ntohl(th->seq), > conntrack->proto.tcp.seen[dir].td_end)) { > /* Attempt to reopen a closed connection. > * Delete this connection and look up again. */ > write_unlock_bh(&tcp_lock); > if (del_timer(&conntrack->timeout)) > conntrack->timeout.function((unsigned long) > conntrack); > return -NF_REPEAT; > } else { > write_unlock_bh(&tcp_lock); > if (LOG_INVALID(IPPROTO_TCP)) > nf_log_packet(pf, 0, skb, NULL, NULL, > NULL, "nf_ct_tcp: invalid SYN"); > return -NF_ACCEPT; > } > --- cut here --- > > It seems that tcp_conntracks allows such (TCP_CONNTRACK_TIME_WAIT -> > TCP_CONNTRACK_SYN_SENT) transition, pointing to rfc1122: > > When a connection is closed actively, it MUST linger in > TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). > However, it MAY accept a new SYN from the remote TCP to > reopen the connection directly from TIME-WAIT state, if it: > > (1) assigns its initial sequence number for the new > connection to be larger than the largest sequence > number it used on the previous connection incarnation, > and > > (2) returns to TIME-WAIT state if the SYN turns out to be > an old duplicate. > > So, it seems that this "after(...)" does not match this packet, right? > > Patric, what do you think about this? Yes, that seems to be the case. Could you post a tcpdump (and add a few printks printing the td_end and the flag values of both directions if possible)? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-03 13:06 ` Krzysztof Oledzki 2007-10-03 13:54 ` Jozsef Kadlecsik 2007-10-04 6:15 ` Patrick McHardy @ 2007-10-10 19:26 ` Jozsef Kadlecsik 2007-10-11 4:04 ` Patrick McHardy ` (2 more replies) 2 siblings, 3 replies; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-10 19:26 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Patrick McHardy Hi, On Wed, 3 Oct 2007, Krzysztof Oledzki wrote: > I also wondering if the code from nf_conntrack_proto_tcp.c is correct: > > --- cut here --- > new_state = tcp_conntracks[dir][index][old_state]; > > switch (new_state) > (...) > case TCP_CONNTRACK_SYN_SENT: > if (old_state < TCP_CONNTRACK_TIME_WAIT) > break; > if ((conntrack->proto.tcp.seen[dir].flags & > IP_CT_TCP_FLAG_CLOSE_INIT) > || after(ntohl(th->seq), > conntrack->proto.tcp.seen[dir].td_end)) { > /* Attempt to reopen a closed connection. > * Delete this connection and look up again. */ > write_unlock_bh(&tcp_lock); > if (del_timer(&conntrack->timeout)) > conntrack->timeout.function((unsigned long) > conntrack); > return -NF_REPEAT; With your description I could reproduce the bug and actually you were completely right: the code above is incorrect. Somehow I was able to misread RFC1122 and mixed the roles :-(: When a connection is >>closed actively<<, it MUST linger in TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). However, it MAY >>accept<< a new SYN from the remote TCP to reopen the connection directly from TIME-WAIT state, if it: [...] The fix is as follows: if the receiver initiated an active close, then the sender may reopen the connection - otherwise try to figure out if we hold a dead connection. Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c index eb3fe74..1b836d0 100644 --- a/net/netfilter/nf_conntrack_proto_tcp.c +++ b/net/netfilter/nf_conntrack_proto_tcp.c @@ -831,6 +831,20 @@ static int tcp_packet(struct nf_conn *conntrack, tuple = &conntrack->tuplehash[dir].tuple; switch (new_state) { + case TCP_CONNTRACK_SYN_SENT: + if (old_state < TCP_CONNTRACK_TIME_WAIT) + break; + if (conntrack->proto.tcp.seen[!dir].flags & + IP_CT_TCP_FLAG_CLOSE_INIT) { + /* Attempt to reopen a closed connection. + * Delete this connection and look up again. */ + write_unlock_bh(&tcp_lock); + if (del_timer(&conntrack->timeout)) + conntrack->timeout.function((unsigned long) + conntrack); + return -NF_REPEAT; + } + /* Fall through */ case TCP_CONNTRACK_IGNORE: /* Ignored packets: * @@ -879,27 +893,6 @@ static int tcp_packet(struct nf_conn *conntrack, nf_log_packet(pf, 0, skb, NULL, NULL, NULL, "nf_ct_tcp: invalid state "); return -NF_ACCEPT; - case TCP_CONNTRACK_SYN_SENT: - if (old_state < TCP_CONNTRACK_TIME_WAIT) - break; - if ((conntrack->proto.tcp.seen[dir].flags & - IP_CT_TCP_FLAG_CLOSE_INIT) - || after(ntohl(th->seq), - conntrack->proto.tcp.seen[dir].td_end)) { - /* Attempt to reopen a closed connection. - * Delete this connection and look up again. */ - write_unlock_bh(&tcp_lock); - if (del_timer(&conntrack->timeout)) - conntrack->timeout.function((unsigned long) - conntrack); - return -NF_REPEAT; - } else { - write_unlock_bh(&tcp_lock); - if (LOG_INVALID(IPPROTO_TCP)) - nf_log_packet(pf, 0, skb, NULL, NULL, - NULL, "nf_ct_tcp: invalid SYN"); - return -NF_ACCEPT; - } case TCP_CONNTRACK_CLOSE: if (index == TCP_RST_SET && ((test_bit(IPS_SEEN_REPLY_BIT, &conntrack->status) Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-10 19:26 ` Jozsef Kadlecsik @ 2007-10-11 4:04 ` Patrick McHardy 2007-10-11 13:52 ` Krzysztof Oledzki 2007-10-11 13:50 ` Krzysztof Oledzki [not found] ` <Pine.LNX.4.64.0710112238010.29814@bizon.gios.gov.pl> 2 siblings, 1 reply; 40+ messages in thread From: Patrick McHardy @ 2007-10-11 4:04 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List Jozsef Kadlecsik wrote: > With your description I could reproduce the bug and actually you were > completely right: the code above is incorrect. Somehow I was able to > misread RFC1122 and mixed the roles :-(: > > When a connection is >>closed actively<<, it MUST linger in > TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). > However, it MAY >>accept<< a new SYN from the remote TCP to > reopen the connection directly from TIME-WAIT state, if it: > [...] > > The fix is as follows: if the receiver initiated an active close, then the > sender may reopen the connection - otherwise try to figure out if we hold > a dead connection. > > Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> Applied, thanks Jozsef. I'll push this to -stable once it hits upstream. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-11 4:04 ` Patrick McHardy @ 2007-10-11 13:52 ` Krzysztof Oledzki 2007-10-31 19:34 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-11 13:52 UTC (permalink / raw) To: Patrick McHardy; +Cc: Jozsef Kadlecsik, Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 1052 bytes --] On Thu, 11 Oct 2007, Patrick McHardy wrote: > Jozsef Kadlecsik wrote: >> With your description I could reproduce the bug and actually you were >> completely right: the code above is incorrect. Somehow I was able to >> misread RFC1122 and mixed the roles :-(: >> >> When a connection is >>closed actively<<, it MUST linger in >> TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). >> However, it MAY >>accept<< a new SYN from the remote TCP to >> reopen the connection directly from TIME-WAIT state, if it: >> [...] >> >> The fix is as follows: if the receiver initiated an active close, then the >> sender may reopen the connection - otherwise try to figure out if we hold a >> dead connection. >> >> Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> > > > Applied, thanks Jozsef. I'll push this to -stable once it hits upstream. > It would be nice to have this fixed in both 2.6.22-stable and 2.6.23-stable if that is possible. Thank you. Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-11 13:52 ` Krzysztof Oledzki @ 2007-10-31 19:34 ` Krzysztof Oledzki 2007-11-05 11:11 ` Patrick McHardy 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-31 19:34 UTC (permalink / raw) To: Patrick McHardy Cc: Jozsef Kadlecsik, Netfilter Developer Mailing List, stable [-- Attachment #1: Type: TEXT/PLAIN, Size: 1536 bytes --] On Thu, 11 Oct 2007, Krzysztof Oledzki wrote: > > > On Thu, 11 Oct 2007, Patrick McHardy wrote: > >> Jozsef Kadlecsik wrote: >>> With your description I could reproduce the bug and actually you were >>> completely right: the code above is incorrect. Somehow I was able to >>> misread RFC1122 and mixed the roles :-(: >>> >>> When a connection is >>closed actively<<, it MUST linger in >>> TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). >>> However, it MAY >>accept<< a new SYN from the remote TCP to >>> reopen the connection directly from TIME-WAIT state, if it: >>> [...] >>> >>> The fix is as follows: if the receiver initiated an active close, then the >>> sender may reopen the connection - otherwise try to figure out if we hold >>> a dead connection. >>> >>> Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> >> >> >> Applied, thanks Jozsef. I'll push this to -stable once it hits upstream. >> > > It would be nice to have this fixed in both 2.6.22-stable and 2.6.23-stable > if that is possible. Thank you. It seems that this patch together with the additional fix did not get pushed to -stable. Any chances to add those two patches to 2.6.22.11: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=17311393f969090ab060540bd9dbe7dc885a76d5 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=bc34b841556aad437baf4199744e55500bfa2088 ? Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-31 19:34 ` Krzysztof Oledzki @ 2007-11-05 11:11 ` Patrick McHardy 2007-11-14 21:46 ` [stable] " Greg KH 0 siblings, 1 reply; 40+ messages in thread From: Patrick McHardy @ 2007-11-05 11:11 UTC (permalink / raw) To: Krzysztof Oledzki Cc: Jozsef Kadlecsik, Netfilter Developer Mailing List, stable Krzysztof Oledzki wrote: >> It would be nice to have this fixed in both 2.6.22-stable and >> 2.6.23-stable if that is possible. Thank you. > > It seems that this patch together with the additional fix did not get > pushed to -stable. Any chances to add those two patches to 2.6.22.11: Sorry for being so late. I'll send them to -stable, but I'm not sure if there will be another 2.6.22 release. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [stable] PATCH: "invalid SYNIN=" - a patch and a question 2007-11-05 11:11 ` Patrick McHardy @ 2007-11-14 21:46 ` Greg KH 0 siblings, 0 replies; 40+ messages in thread From: Greg KH @ 2007-11-14 21:46 UTC (permalink / raw) To: Patrick McHardy Cc: Krzysztof Oledzki, Netfilter Developer Mailing List, stable, Jozsef Kadlecsik On Mon, Nov 05, 2007 at 12:11:12PM +0100, Patrick McHardy wrote: > Krzysztof Oledzki wrote: > >> It would be nice to have this fixed in both 2.6.22-stable and > >> 2.6.23-stable if that is possible. Thank you. > > > > It seems that this patch together with the additional fix did not get > > pushed to -stable. Any chances to add those two patches to 2.6.22.11: > > > Sorry for being so late. I'll send them to -stable, but I'm not sure if > there will be another 2.6.22 release. Yes there will be, so feel free to send me any patches for the .22 series if you want them to go into there. thanks, greg k-h ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-10 19:26 ` Jozsef Kadlecsik 2007-10-11 4:04 ` Patrick McHardy @ 2007-10-11 13:50 ` Krzysztof Oledzki [not found] ` <Pine.LNX.4.64.0710112238010.29814@bizon.gios.gov.pl> 2 siblings, 0 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-11 13:50 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Netfilter Developer Mailing List, Patrick McHardy [-- Attachment #1: Type: TEXT/PLAIN, Size: 1800 bytes --] On Wed, 10 Oct 2007, Jozsef Kadlecsik wrote: > Hi, > > On Wed, 3 Oct 2007, Krzysztof Oledzki wrote: > >> I also wondering if the code from nf_conntrack_proto_tcp.c is correct: >> >> --- cut here --- >> new_state = tcp_conntracks[dir][index][old_state]; >> >> switch (new_state) >> (...) >> case TCP_CONNTRACK_SYN_SENT: >> if (old_state < TCP_CONNTRACK_TIME_WAIT) >> break; >> if ((conntrack->proto.tcp.seen[dir].flags & >> IP_CT_TCP_FLAG_CLOSE_INIT) >> || after(ntohl(th->seq), >> conntrack->proto.tcp.seen[dir].td_end)) { >> /* Attempt to reopen a closed connection. >> * Delete this connection and look up again. */ >> write_unlock_bh(&tcp_lock); >> if (del_timer(&conntrack->timeout)) >> conntrack->timeout.function((unsigned long) >> conntrack); >> return -NF_REPEAT; > > With your description I could reproduce the bug and actually you were > completely right: the code above is incorrect. Somehow I was able to > misread RFC1122 and mixed the roles :-(: > > When a connection is >>closed actively<<, it MUST linger in > TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). > However, it MAY >>accept<< a new SYN from the remote TCP to > reopen the connection directly from TIME-WAIT state, if it: > [...] > > The fix is as follows: if the receiver initiated an active close, then the > sender may reopen the connection - otherwise try to figure out if we hold > a dead connection. > > Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> Thank you, it seems that this patch solved my problem. Tested-by: Krzysztof Piotr Oledzki <ole@ans.pl> Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
[parent not found: <Pine.LNX.4.64.0710112238010.29814@bizon.gios.gov.pl>]
* Re: PATCH: "invalid SYNIN=" - a patch and a question [not found] ` <Pine.LNX.4.64.0710112238010.29814@bizon.gios.gov.pl> @ 2007-10-11 22:42 ` Jozsef Kadlecsik 2007-10-12 10:09 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-11 22:42 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Patrick McHardy On Thu, 11 Oct 2007, Krzysztof Oledzki wrote: > Something is still wrong in this area. With the recent fix there are no > more "invalid SYNIN=" messages, but with unsuccessful request simulated > with "echo|nc" next connections are not possible. [...] > --- cut here --- > sysctl net.ipv4.ip_local_port_range="50000 50003" > sysctl net.netfilter.nf_conntrack_log_invalid=255 > > while true ; do echo|nc -w 1 wp.pl 80 2>/dev/null >/dev/null; echo -ne "HEAD / > HTTP/1.0\r\nHost: www.wp.pl\r\n\r\n"|nc -w 2 wp.pl 80 ; sleep 1 ; done > --- cut here --- Odd, but I could not reproduce it using my local test webserver. However looking through the tcpdump and the kernel log, I think the problem is as follows: the aborted connection still lingers in conntrack for 10s (in order to catch late packets) and reopening after RST is not handled properly yet. You are digging into the grey areas of TCP conntrack ;-). Could you check the patch below? If one side aborts an established connection, the entry still lingers for 10s in conntrack for the late packets. Allow to open up the connection again for the party which sent the RST packet. Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> --- net/netfilter/nf_conntrack_proto_tcp.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c index 1b836d0..10a04ff 100644 --- a/net/netfilter/nf_conntrack_proto_tcp.c +++ b/net/netfilter/nf_conntrack_proto_tcp.c @@ -834,9 +834,11 @@ static int tcp_packet(struct nf_conn *conntrack, case TCP_CONNTRACK_SYN_SENT: if (old_state < TCP_CONNTRACK_TIME_WAIT) break; - if (conntrack->proto.tcp.seen[!dir].flags & - IP_CT_TCP_FLAG_CLOSE_INIT) { - /* Attempt to reopen a closed connection. + if ((conntrack->proto.tcp.seen[!dir].flags & + IP_CT_TCP_FLAG_CLOSE_INIT) + || (conntrack->proto.tcp.last_dir == dir + && conntrack->proto.tcp.last_index == TCP_RST_SET)) { + /* Attempt to reopen a closed/aborted connection. * Delete this connection and look up again. */ write_unlock_bh(&tcp_lock); if (del_timer(&conntrack->timeout)) -- 1.5.3.4 Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-11 22:42 ` Jozsef Kadlecsik @ 2007-10-12 10:09 ` Krzysztof Oledzki 2007-10-12 14:48 ` Jozsef Kadlecsik 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-12 10:09 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Netfilter Developer Mailing List, Patrick McHardy [-- Attachment #1: Type: TEXT/PLAIN, Size: 1396 bytes --] On Fri, 12 Oct 2007, Jozsef Kadlecsik wrote: > On Thu, 11 Oct 2007, Krzysztof Oledzki wrote: > >> Something is still wrong in this area. With the recent fix there are no >> more "invalid SYNIN=" messages, but with unsuccessful request simulated >> with "echo|nc" next connections are not possible. > [...] >> --- cut here --- >> sysctl net.ipv4.ip_local_port_range="50000 50003" >> sysctl net.netfilter.nf_conntrack_log_invalid=255 >> >> while true ; do echo|nc -w 1 wp.pl 80 2>/dev/null >/dev/null; echo -ne "HEAD / >> HTTP/1.0\r\nHost: www.wp.pl\r\n\r\n"|nc -w 2 wp.pl 80 ; sleep 1 ; done >> --- cut here --- > > Odd, but I could not reproduce it using my local test webserver. I think you may test it with wp.pl, it is quite a big portal. No harm will be done with little more requests. ;) > However looking through the tcpdump and the kernel log, I think the > problem is as follows: the aborted connection still lingers in conntrack > for 10s (in order to catch late packets) and reopening after RST is not > handled properly yet. You are digging into the grey areas of TCP > conntrack ;-). > > Could you check the patch below? Assuming that I did not mispatched my kernel, this patch does not solve described problem. :( Do you need more tcpdumps or maybe a "conntrack -o time -E" log? Thank you. Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-12 10:09 ` Krzysztof Oledzki @ 2007-10-12 14:48 ` Jozsef Kadlecsik 2007-10-12 18:09 ` Krzysztof Oledzki 0 siblings, 1 reply; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-12 14:48 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Patrick McHardy On Fri, 12 Oct 2007, Krzysztof Oledzki wrote: > > Odd, but I could not reproduce it using my local test webserver. > > I think you may test it with wp.pl, it is quite a big portal. No harm will be > done with little more requests. ;) All of the machines I have got access are behind firewalls. > > Could you check the patch below? > > Assuming that I did not mispatched my kernel, this patch does not solve > described problem. :( The patch should have been applied on the top of the previous one I sent. > Do you need more tcpdumps or maybe a "conntrack -o time -E" log? Could you send me a pcap file captured with '-s 0'? Then I could replay it and see what happens. Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-12 14:48 ` Jozsef Kadlecsik @ 2007-10-12 18:09 ` Krzysztof Oledzki 2007-10-17 17:41 ` Jozsef Kadlecsik 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-12 18:09 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Netfilter Developer Mailing List, Patrick McHardy [-- Attachment #1: Type: TEXT/PLAIN, Size: 1524 bytes --] On Fri, 12 Oct 2007, Jozsef Kadlecsik wrote: > On Fri, 12 Oct 2007, Krzysztof Oledzki wrote: > >>> Odd, but I could not reproduce it using my local test webserver. >> >> I think you may test it with wp.pl, it is quite a big portal. No harm will be >> done with little more requests. ;) > > All of the machines I have got access are behind firewalls. Oh, I see. >>> Could you check the patch below? >> >> Assuming that I did not mispatched my kernel, this patch does not solve >> described problem. :( > > The patch should have been applied on the top of the previous one I sent. OK. I did it exactly this way. >> Do you need more tcpdumps or maybe a "conntrack -o time -E" log? > > Could you send me a pcap file captured with '-s 0'? Then I could replay it > and see what happens. Attached. Oct 12 19:52:12 wrestler kernel: nf_ct_tcp: invalid packed ignored IN= OUT= SRC=195.177.210.7 DST=212.77.100.101 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57209 DF PROTO=TCP SPT=50001 DPT=80 SEQ=4171859972 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01C5577D0000000001030306) UID=0 Oct 12 19:52:12 wrestler kernel: nf_ct_tcp: killing out of sync session IN= OUT= SRC=212.77.100.101 DST=195.177.210.7 LEN=60 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=80 DPT=50001 SEQ=3512463029 ACK=4171859973 WINDOW=5792 RES=0x00 ACK SYN URGP=0 OPT (020405B40402080A2261ACA501C5577D01030302) Please let me know if you need more data. Thank you. Best regards, Krzysztof Olędzki [-- Attachment #2: Type: APPLICATION/octet-stream, Size: 1535 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-12 18:09 ` Krzysztof Oledzki @ 2007-10-17 17:41 ` Jozsef Kadlecsik 2007-10-17 20:08 ` Krzysztof Oledzki 2007-10-18 8:43 ` Patrick McHardy 0 siblings, 2 replies; 40+ messages in thread From: Jozsef Kadlecsik @ 2007-10-17 17:41 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List, Patrick McHardy On Fri, 12 Oct 2007, Krzysztof Oledzki wrote: > > Could you send me a pcap file captured with '-s 0'? Then I could replay it > > and see what happens. > > Attached. Thank you, that was really useful. So instead of the patch I sent last time about active close, please apply the patch below: setting the last_dir parameter was missing from the previous one. Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c index 1b836d0..67b39b8 100644 --- a/net/netfilter/nf_conntrack_proto_tcp.c +++ b/net/netfilter/nf_conntrack_proto_tcp.c @@ -834,9 +834,11 @@ static int tcp_packet(struct nf_conn *conntrack, case TCP_CONNTRACK_SYN_SENT: if (old_state < TCP_CONNTRACK_TIME_WAIT) break; - if (conntrack->proto.tcp.seen[!dir].flags & - IP_CT_TCP_FLAG_CLOSE_INIT) { - /* Attempt to reopen a closed connection. + if ((conntrack->proto.tcp.seen[!dir].flags & + IP_CT_TCP_FLAG_CLOSE_INIT) + || (conntrack->proto.tcp.last_dir == dir + && conntrack->proto.tcp.last_index == TCP_RST_SET)) { + /* Attempt to reopen a closed/aborted connection. * Delete this connection and look up again. */ write_unlock_bh(&tcp_lock); if (del_timer(&conntrack->timeout)) @@ -925,6 +927,7 @@ static int tcp_packet(struct nf_conn *conntrack, in_window: /* From now on we have got in-window packets */ conntrack->proto.tcp.last_index = index; + conntrack->proto.tcp.last_dir = dir; pr_debug("tcp_conntracks: "); NF_CT_DUMP_TUPLE(tuple); Best regards, Jozsef - E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt Address : KFKI Research Institute for Particle and Nuclear Physics H-1525 Budapest 114, POB. 49, Hungary ^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-17 17:41 ` Jozsef Kadlecsik @ 2007-10-17 20:08 ` Krzysztof Oledzki 2007-10-18 8:43 ` Patrick McHardy 1 sibling, 0 replies; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-17 20:08 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Netfilter Developer Mailing List, Patrick McHardy [-- Attachment #1: Type: TEXT/PLAIN, Size: 614 bytes --] On Wed, 17 Oct 2007, Jozsef Kadlecsik wrote: > On Fri, 12 Oct 2007, Krzysztof Oledzki wrote: > >>> Could you send me a pcap file captured with '-s 0'? Then I could replay it >>> and see what happens. >> >> Attached. > > Thank you, that was really useful. So instead of the patch I sent last > time about active close, please apply the patch below: setting the > last_dir parameter was missing from the previous one. > > Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> Thank you very much. Tested-by: Krzysztof Piotr Oledzki <ole@ans.pl> Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-17 17:41 ` Jozsef Kadlecsik 2007-10-17 20:08 ` Krzysztof Oledzki @ 2007-10-18 8:43 ` Patrick McHardy 1 sibling, 0 replies; 40+ messages in thread From: Patrick McHardy @ 2007-10-18 8:43 UTC (permalink / raw) To: Jozsef Kadlecsik; +Cc: Krzysztof Oledzki, Netfilter Developer Mailing List Jozsef Kadlecsik wrote: > On Fri, 12 Oct 2007, Krzysztof Oledzki wrote: > >>> Could you send me a pcap file captured with '-s 0'? Then I could replay it >>> and see what happens. >> Attached. > > Thank you, that was really useful. So instead of the patch I sent last > time about active close, please apply the patch below: setting the > last_dir parameter was missing from the previous one. > > Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> Applied, thanks Jozsef. I'll push the combined patch to -stable. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-09-26 12:27 PATCH: "invalid SYNIN=" - a patch and a question Krzysztof Oledzki 2007-10-03 13:06 ` Krzysztof Oledzki @ 2007-10-04 5:45 ` Patrick McHardy 2007-10-17 20:13 ` Krzysztof Oledzki 1 sibling, 1 reply; 40+ messages in thread From: Patrick McHardy @ 2007-10-04 5:45 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List Krzysztof Oledzki wrote: > [NETFILTER]: Trivial fix for "nf_ct_tcp: invalid SYN" > > This nf_log_packet needs a space char. Trivial fix > > Signed-off-by: Krzysztof Piotr Oledzki <ole@ans.pl> Applied, thanks. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-04 5:45 ` Patrick McHardy @ 2007-10-17 20:13 ` Krzysztof Oledzki 2007-10-18 8:43 ` Patrick McHardy 0 siblings, 1 reply; 40+ messages in thread From: Krzysztof Oledzki @ 2007-10-17 20:13 UTC (permalink / raw) To: Patrick McHardy; +Cc: Netfilter Developer Mailing List [-- Attachment #1: Type: TEXT/PLAIN, Size: 444 bytes --] On Thu, 4 Oct 2007, Patrick McHardy wrote: > Krzysztof Oledzki wrote: >> [NETFILTER]: Trivial fix for "nf_ct_tcp: invalid SYN" >> >> This nf_log_packet needs a space char. Trivial fix >> >> Signed-off-by: Krzysztof Piotr Oledzki <ole@ans.pl> > > Applied, thanks. With latest fixes from Jozsef this patch should be dropped as this code was removed and now there is nothing to fix. ;) Best regards, Krzysztof Olędzki ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: PATCH: "invalid SYNIN=" - a patch and a question 2007-10-17 20:13 ` Krzysztof Oledzki @ 2007-10-18 8:43 ` Patrick McHardy 0 siblings, 0 replies; 40+ messages in thread From: Patrick McHardy @ 2007-10-18 8:43 UTC (permalink / raw) To: Krzysztof Oledzki; +Cc: Netfilter Developer Mailing List Krzysztof Oledzki wrote: > > > On Thu, 4 Oct 2007, Patrick McHardy wrote: > >> Krzysztof Oledzki wrote: >>> [NETFILTER]: Trivial fix for "nf_ct_tcp: invalid SYN" >>> >>> This nf_log_packet needs a space char. Trivial fix >>> >>> Signed-off-by: Krzysztof Piotr Oledzki <ole@ans.pl> >> >> Applied, thanks. > > With latest fixes from Jozsef this patch should be dropped as this code > was removed and now there is nothing to fix. ;) I already dropped it when I applied the first fix since it included that part :) ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2007-11-14 21:56 UTC | newest]
Thread overview: 40+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-09-26 12:27 PATCH: "invalid SYNIN=" - a patch and a question Krzysztof Oledzki
2007-10-03 13:06 ` Krzysztof Oledzki
2007-10-03 13:54 ` Jozsef Kadlecsik
2007-10-03 19:57 ` Krzysztof Oledzki
2007-10-04 7:26 ` Jozsef Kadlecsik
2007-10-04 5:59 ` Patrick McHardy
2007-10-04 7:52 ` Jozsef Kadlecsik
2007-10-04 7:59 ` Patrick McHardy
2007-10-04 8:41 ` Jozsef Kadlecsik
2007-10-05 20:06 ` Krzysztof Oledzki
2007-10-08 9:32 ` Jozsef Kadlecsik
2007-10-08 9:45 ` Jozsef Kadlecsik
2007-10-08 13:56 ` Krzysztof Oledzki
2007-10-08 14:01 ` Krzysztof Oledzki
2007-10-08 13:58 ` Krzysztof Oledzki
2007-10-08 14:01 ` Jozsef Kadlecsik
2007-10-08 17:28 ` Krzysztof Oledzki
2007-10-09 8:11 ` Jozsef Kadlecsik
2007-10-09 13:17 ` Krzysztof Oledzki
2007-10-09 13:22 ` Krzysztof Oledzki
2007-10-08 16:39 ` Patrick McHardy
2007-10-08 17:23 ` Krzysztof Oledzki
2007-10-04 6:15 ` Patrick McHardy
2007-10-10 19:26 ` Jozsef Kadlecsik
2007-10-11 4:04 ` Patrick McHardy
2007-10-11 13:52 ` Krzysztof Oledzki
2007-10-31 19:34 ` Krzysztof Oledzki
2007-11-05 11:11 ` Patrick McHardy
2007-11-14 21:46 ` [stable] " Greg KH
2007-10-11 13:50 ` Krzysztof Oledzki
[not found] ` <Pine.LNX.4.64.0710112238010.29814@bizon.gios.gov.pl>
2007-10-11 22:42 ` Jozsef Kadlecsik
2007-10-12 10:09 ` Krzysztof Oledzki
2007-10-12 14:48 ` Jozsef Kadlecsik
2007-10-12 18:09 ` Krzysztof Oledzki
2007-10-17 17:41 ` Jozsef Kadlecsik
2007-10-17 20:08 ` Krzysztof Oledzki
2007-10-18 8:43 ` Patrick McHardy
2007-10-04 5:45 ` Patrick McHardy
2007-10-17 20:13 ` Krzysztof Oledzki
2007-10-18 8:43 ` Patrick McHardy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).