* Whence a description of how to enable TCP FASTOPEN in a net-next kernel?
@ 2012-12-17 21:27 Rick Jones
2012-12-17 21:56 ` Eric Dumazet
0 siblings, 1 reply; 4+ messages in thread
From: Rick Jones @ 2012-12-17 21:27 UTC (permalink / raw)
To: netdev
Is there a writeup describing the steps needed to enable TCP_FASTOPEN in
a net-next kernel? (pulled earlier today)
I am looking to debug netperf's support for enabling the feature and I
want to make sure I've enabled things correctly in the kernel. Thusfar
I've set the tcp_fastopen sysctl to one, and I see the "client" side of
netperf making the appropriate sendto() call, and I see what appears to
be the correct setsockopt being set on the server side, but my tcpdump
traces of the traffic flowing over loopback in my test setup, while
showing the client including the experimental option, do not show the
server side responding:
13:10:23.870202 IP localhost.5923 > localhost.54363: Flags [S], seq
935361110, win 43690, options [mss 65495,sackOK,TS val 889762 ecr
0,nop,wscale 7,Unknown Option 254f989], length 0
13:10:23.870214 IP localhost.54363 > localhost.5923: Flags [S.], seq
4210640362, ack 935361111, win 43690, options [mss 65495,sackOK,TS val
889762 ecr 889762,nop,wscale 7], length 0
The netserver side strace snippet:
3861 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
3861 getsockopt(8, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
3861 getsockopt(8, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
3861 setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
3861 bind(8, {sa_family=AF_INET, sin_port=htons(0),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
3861 setsockopt(8, SOL_TCP, 0x17 /* TCP_??? */, [5], 4) = 0
3861 listen(8, 5) = 0
...
3861 accept(8, {sa_family=AF_INET, sin_port=htons(5923),
sin_addr=inet_addr("127.0.0.1")}, [16]) = 9
3861 recvfrom(9, "n", 1, 0, NULL, NULL) = 1
3861 sendto(9, "n", 1, 0, NULL, 0) = 1
3861 getsockopt(9, SOL_SOCKET, SO_RCVBUF, [262030], [4]) = 0
3861 getsockopt(9, SOL_SOCKET, SO_SNDBUF, [663750], [4]) = 0
3861 close(9) = 0
lather, rinse, repeat the accept sequence off that listen endpoint.
happy benchmarking,
rick jones
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Whence a description of how to enable TCP FASTOPEN in a net-next kernel? 2012-12-17 21:27 Whence a description of how to enable TCP FASTOPEN in a net-next kernel? Rick Jones @ 2012-12-17 21:56 ` Eric Dumazet 2012-12-17 22:03 ` Eric Dumazet 2012-12-17 22:56 ` Rick Jones 0 siblings, 2 replies; 4+ messages in thread From: Eric Dumazet @ 2012-12-17 21:56 UTC (permalink / raw) To: Rick Jones; +Cc: netdev On Mon, 2012-12-17 at 13:27 -0800, Rick Jones wrote: > Is there a writeup describing the steps needed to enable TCP_FASTOPEN in > a net-next kernel? (pulled earlier today) > > I am looking to debug netperf's support for enabling the feature and I > want to make sure I've enabled things correctly in the kernel. Thusfar > I've set the tcp_fastopen sysctl to one, and I see the "client" side of > netperf making the appropriate sendto() call, and I see what appears to > be the correct setsockopt being set on the server side, but my tcpdump > traces of the traffic flowing over loopback in my test setup, while > showing the client including the experimental option, do not show the > server side responding: > > 13:10:23.870202 IP localhost.5923 > localhost.54363: Flags [S], seq > 935361110, win 43690, options [mss 65495,sackOK,TS val 889762 ecr > 0,nop,wscale 7,Unknown Option 254f989], length 0 > 13:10:23.870214 IP localhost.54363 > localhost.5923: Flags [S.], seq > 4210640362, ack 935361111, win 43690, options [mss 65495,sackOK,TS val > 889762 ecr 889762,nop,wscale 7], length 0 > > The netserver side strace snippet: > > 3861 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 > 3861 getsockopt(8, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0 > 3861 getsockopt(8, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0 > 3861 setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > 3861 bind(8, {sa_family=AF_INET, sin_port=htons(0), > sin_addr=inet_addr("0.0.0.0")}, 16) = 0 > 3861 setsockopt(8, SOL_TCP, 0x17 /* TCP_??? */, [5], 4) = 0 > 3861 listen(8, 5) = 0 > ... > 3861 accept(8, {sa_family=AF_INET, sin_port=htons(5923), > sin_addr=inet_addr("127.0.0.1")}, [16]) = 9 > 3861 recvfrom(9, "n", 1, 0, NULL, NULL) = 1 > 3861 sendto(9, "n", 1, 0, NULL, 0) = 1 > 3861 getsockopt(9, SOL_SOCKET, SO_RCVBUF, [262030], [4]) = 0 > 3861 getsockopt(9, SOL_SOCKET, SO_SNDBUF, [663750], [4]) = 0 > 3861 close(9) = 0 > lather, rinse, repeat the accept sequence off that listen endpoint. > > happy benchmarking, I guess you need to enable fastopen both for client and server : echo 3 >/proc/sys//net/ipv4/tcp_fastopen ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Whence a description of how to enable TCP FASTOPEN in a net-next kernel? 2012-12-17 21:56 ` Eric Dumazet @ 2012-12-17 22:03 ` Eric Dumazet 2012-12-17 22:56 ` Rick Jones 1 sibling, 0 replies; 4+ messages in thread From: Eric Dumazet @ 2012-12-17 22:03 UTC (permalink / raw) To: Rick Jones; +Cc: netdev On Mon, 2012-12-17 at 13:56 -0800, Eric Dumazet wrote: > > I guess you need to enable fastopen both for client and server : > > echo 3 >/proc/sys//net/ipv4/tcp_fastopen vi +475 Documentation/networking/ip-sysctl.txt tcp_fastopen - INTEGER Enable TCP Fast Open feature (draft-ietf-tcpm-fastopen) to send data in the opening SYN packet. To use this feature, the client application must use sendmsg() or sendto() with MSG_FASTOPEN flag rather than connect() to perform a TCP handshake automatically. The values (bitmap) are 1: Enables sending data in the opening SYN on the client. 2: Enables TCP Fast Open on the server side, i.e., allowing data in a SYN packet to be accepted and passed to the application before 3-way hand shake finishes. 4: Send data in the opening SYN regardless of cookie availability and without a cookie option. 0x100: Accept SYN data w/o validating the cookie. 0x200: Accept data-in-SYN w/o any cookie option present. 0x400/0x800: Enable Fast Open on all listeners regardless of the TCP_FASTOPEN socket option. The two different flags designate two different ways of setting max_qlen without the TCP_FASTOPEN socket option. Default: 0 Note that the client & server side Fast Open flags (1 and 2 respectively) must be also enabled before the rest of flags can take effect. See include/net/tcp.h and the code for more details. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Whence a description of how to enable TCP FASTOPEN in a net-next kernel? 2012-12-17 21:56 ` Eric Dumazet 2012-12-17 22:03 ` Eric Dumazet @ 2012-12-17 22:56 ` Rick Jones 1 sibling, 0 replies; 4+ messages in thread From: Rick Jones @ 2012-12-17 22:56 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev On 12/17/2012 01:56 PM, Eric Dumazet wrote: > On Mon, 2012-12-17 at 13:27 -0800, Rick Jones wrote: >> [an explanation of what he'd done that hadn't worked] > > I guess you need to enable fastopen both for client and server : > > echo 3 >/proc/sys//net/ipv4/tcp_fastopen Looks like I'm good now: raj@tardy-ubuntu-1204:~$ cat /proc/sys/net/ipv4/tcp_fastopen 3 raj@tardy-ubuntu-1204:~$ sudo tcpdump -c 30 -i lo 'not port 12865' [sudo] password for raj: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes 14:28:43.536166 IP localhost.29105 > localhost.srvr: Flags [S], seq 378007190, win 43690, options [mss 65495,sackOK,TS val 4294942097 ecr 0,nop,wscale 7,Unknown Option 254f989], length 0 14:28:43.536191 IP localhost.srvr > localhost.29105: Flags [S.], seq 2030806688, ack 378007191, win 43690, options [mss 65495,sackOK,TS val 4294942097 ecr 4294942097,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], length 0 14:28:43.537421 IP localhost.29105 > localhost.srvr: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942097], length 1 14:28:43.537445 IP localhost.srvr > localhost.29105: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.537525 IP localhost.srvr > localhost.29105: Flags [P.], seq 1:2, ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 1 14:28:43.537542 IP localhost.srvr > localhost.29105: Flags [F.], seq 2, ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.537727 IP localhost.29105 > localhost.srvr: Flags [F.], seq 2, ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.537741 IP localhost.srvr > localhost.29105: Flags [.], ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.537895 IP localhost.29106 > localhost.srvr: Flags [S], seq 1735077945:1735077946, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 0,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], length 1 14:28:43.537909 IP localhost.srvr > localhost.29106: Flags [S.], seq 1983728126, ack 1735077947, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 4294942098,nop,wscale 7], length 0 14:28:43.537924 IP localhost.29106 > localhost.srvr: Flags [.], ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538014 IP localhost.srvr > localhost.29106: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 1 14:28:43.538028 IP localhost.srvr > localhost.29106: Flags [F.], seq 2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538196 IP localhost.29106 > localhost.srvr: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538288 IP localhost.29106 > localhost.srvr: Flags [F.], seq 1, ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538300 IP localhost.srvr > localhost.29106: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538417 IP localhost.29107 > localhost.srvr: Flags [S], seq 3902541042:3902541043, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 0,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], length 1 14:28:43.538431 IP localhost.srvr > localhost.29107: Flags [S.], seq 941945820, ack 3902541044, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 4294942098,nop,wscale 7], length 0 14:28:43.538445 IP localhost.29107 > localhost.srvr: Flags [.], ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538518 IP localhost.srvr > localhost.29107: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 1 14:28:43.538531 IP localhost.srvr > localhost.29107: Flags [F.], seq 2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538654 IP localhost.29107 > localhost.srvr: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538730 IP localhost.29107 > localhost.srvr: Flags [F.], seq 1, ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 14:28:43.538742 IP localhost.srvr > localhost.29107: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0 in which case I suppose that means that netperf top-of-trunk does indeed have client and server side support for TCP_FASTOPEN. Enabled via the test-specific -F option though in the loopback test (in a 1 VCPU VM) I don't see much of a difference (don't suppose I should really): raj@tardy-ubuntu-1204:~/netperf2_trunk/src$ ./netperf -t TCP_CRR -l 30 -i 30,3 -- -F -P ,12345 MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost () port 12345 AF_INET : +/-2.500% @ 99% conf. : demo Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec 16384 87380 1 1 30.00 15909.07 16384 87380 raj@tardy-ubuntu-1204:~/netperf2_trunk/src$ ./netperf -t TCP_CRR -l 30 -i 30,3 -- -P ,12345 MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost () port 12345 AF_INET : +/-2.500% @ 99% conf. : demo Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec 16384 87380 1 1 30.00 15574.37 16384 87380 happy benchmarking, rick jones ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-12-17 22:56 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-12-17 21:27 Whence a description of how to enable TCP FASTOPEN in a net-next kernel? Rick Jones 2012-12-17 21:56 ` Eric Dumazet 2012-12-17 22:03 ` Eric Dumazet 2012-12-17 22:56 ` Rick Jones
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).