Whence a description of how to enable TCP FASTOPEN in a net-next kernel?

Whence a description of how to enable TCP FASTOPEN in a net-next kernel?

[Rick Jones]

Is there a writeup describing the steps needed to enable TCP_FASTOPEN in 
a net-next kernel? (pulled earlier today)

I am looking to debug netperf's support for enabling the feature and I 
want to make sure I've enabled things correctly in the kernel.  Thusfar 
I've set the tcp_fastopen sysctl to one, and I see the "client" side of 
netperf making the appropriate sendto() call, and I see what appears to 
be the correct setsockopt being set on the server side, but my tcpdump 
traces of the traffic flowing over loopback in my test setup, while 
showing the client including the experimental option, do not show the 
server side responding:

13:10:23.870202 IP localhost.5923 > localhost.54363: Flags [S], seq 
935361110, win 43690, options [mss 65495,sackOK,TS val 889762 ecr 
0,nop,wscale 7,Unknown Option 254f989], length 0
13:10:23.870214 IP localhost.54363 > localhost.5923: Flags [S.], seq 
4210640362, ack 935361111, win 43690, options [mss 65495,sackOK,TS val 
889762 ecr 889762,nop,wscale 7], length 0

The netserver side strace snippet:

3861  socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
3861  getsockopt(8, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
3861  getsockopt(8, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
3861  setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
3861  bind(8, {sa_family=AF_INET, sin_port=htons(0), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
3861  setsockopt(8, SOL_TCP, 0x17 /* TCP_??? */, [5], 4) = 0
3861  listen(8, 5)                      = 0
...
3861  accept(8, {sa_family=AF_INET, sin_port=htons(5923), 
sin_addr=inet_addr("127.0.0.1")}, [16]) = 9
3861  recvfrom(9, "n", 1, 0, NULL, NULL) = 1
3861  sendto(9, "n", 1, 0, NULL, 0)     = 1
3861  getsockopt(9, SOL_SOCKET, SO_RCVBUF, [262030], [4]) = 0
3861  getsockopt(9, SOL_SOCKET, SO_SNDBUF, [663750], [4]) = 0
3861  close(9)                          = 0

lather, rinse, repeat the accept sequence off that listen endpoint.

happy benchmarking,

rick jones

Re: Whence a description of how to enable TCP FASTOPEN in a net-next kernel?

[Eric Dumazet]

On Mon, 2012-12-17 at 13:27 -0800, Rick Jones wrote:
> Is there a writeup describing the steps needed to enable TCP_FASTOPEN in 
> a net-next kernel? (pulled earlier today)
> 
> I am looking to debug netperf's support for enabling the feature and I 
> want to make sure I've enabled things correctly in the kernel.  Thusfar 
> I've set the tcp_fastopen sysctl to one, and I see the "client" side of 
> netperf making the appropriate sendto() call, and I see what appears to 
> be the correct setsockopt being set on the server side, but my tcpdump 
> traces of the traffic flowing over loopback in my test setup, while 
> showing the client including the experimental option, do not show the 
> server side responding:
> 
> 13:10:23.870202 IP localhost.5923 > localhost.54363: Flags [S], seq 
> 935361110, win 43690, options [mss 65495,sackOK,TS val 889762 ecr 
> 0,nop,wscale 7,Unknown Option 254f989], length 0
> 13:10:23.870214 IP localhost.54363 > localhost.5923: Flags [S.], seq 
> 4210640362, ack 935361111, win 43690, options [mss 65495,sackOK,TS val 
> 889762 ecr 889762,nop,wscale 7], length 0
> 
> The netserver side strace snippet:
> 
> 3861  socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
> 3861  getsockopt(8, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
> 3861  getsockopt(8, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
> 3861  setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> 3861  bind(8, {sa_family=AF_INET, sin_port=htons(0), 
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> 3861  setsockopt(8, SOL_TCP, 0x17 /* TCP_??? */, [5], 4) = 0
> 3861  listen(8, 5)                      = 0
> ...
> 3861  accept(8, {sa_family=AF_INET, sin_port=htons(5923), 
> sin_addr=inet_addr("127.0.0.1")}, [16]) = 9
> 3861  recvfrom(9, "n", 1, 0, NULL, NULL) = 1
> 3861  sendto(9, "n", 1, 0, NULL, 0)     = 1
> 3861  getsockopt(9, SOL_SOCKET, SO_RCVBUF, [262030], [4]) = 0
> 3861  getsockopt(9, SOL_SOCKET, SO_SNDBUF, [663750], [4]) = 0
> 3861  close(9)                          = 0

> lather, rinse, repeat the accept sequence off that listen endpoint.
> 
> happy benchmarking,

I guess you need to enable fastopen both for client and server :

echo 3 >/proc/sys//net/ipv4/tcp_fastopen

Re: Whence a description of how to enable TCP FASTOPEN in a net-next kernel?

[Eric Dumazet]

On Mon, 2012-12-17 at 13:56 -0800, Eric Dumazet wrote:

> 
> I guess you need to enable fastopen both for client and server :
> 
> echo 3 >/proc/sys//net/ipv4/tcp_fastopen

vi +475 Documentation/networking/ip-sysctl.txt

tcp_fastopen - INTEGER
        Enable TCP Fast Open feature (draft-ietf-tcpm-fastopen) to send data
        in the opening SYN packet. To use this feature, the client application
        must use sendmsg() or sendto() with MSG_FASTOPEN flag rather than
        connect() to perform a TCP handshake automatically.

        The values (bitmap) are
        1: Enables sending data in the opening SYN on the client.
        2: Enables TCP Fast Open on the server side, i.e., allowing data in
           a SYN packet to be accepted and passed to the application before
           3-way hand shake finishes.
        4: Send data in the opening SYN regardless of cookie availability and
           without a cookie option.
        0x100: Accept SYN data w/o validating the cookie.
        0x200: Accept data-in-SYN w/o any cookie option present.
        0x400/0x800: Enable Fast Open on all listeners regardless of the
           TCP_FASTOPEN socket option. The two different flags designate two
           different ways of setting max_qlen without the TCP_FASTOPEN socket
           option.

        Default: 0

        Note that the client & server side Fast Open flags (1 and 2
        respectively) must be also enabled before the rest of flags can take
        effect.

        See include/net/tcp.h and the code for more details.

Re: Whence a description of how to enable TCP FASTOPEN in a net-next kernel?

[Rick Jones]

On 12/17/2012 01:56 PM, Eric Dumazet wrote:
> On Mon, 2012-12-17 at 13:27 -0800, Rick Jones wrote:
>> [an explanation of what he'd done that hadn't worked]
>
> I guess you need to enable fastopen both for client and server :
>
> echo 3 >/proc/sys//net/ipv4/tcp_fastopen


Looks like I'm good now:

raj@tardy-ubuntu-1204:~$ cat /proc/sys/net/ipv4/tcp_fastopen
3
raj@tardy-ubuntu-1204:~$ sudo tcpdump -c 30 -i lo 'not port 12865'
[sudo] password for raj:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
14:28:43.536166 IP localhost.29105 > localhost.srvr: Flags [S], seq 
378007190, win 43690, options [mss 65495,sackOK,TS val 4294942097 ecr 
0,nop,wscale 7,Unknown Option 254f989], length 0
14:28:43.536191 IP localhost.srvr > localhost.29105: Flags [S.], seq 
2030806688, ack 378007191, win 43690, options [mss 65495,sackOK,TS val 
4294942097 ecr 4294942097,nop,wscale 7,Unknown Option 
254f989e73dc061f14d850e], length 0
14:28:43.537421 IP localhost.29105 > localhost.srvr: Flags [P.], seq 
1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942097], 
length 1
14:28:43.537445 IP localhost.srvr > localhost.29105: Flags [.], ack 2, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537525 IP localhost.srvr > localhost.29105: Flags [P.], seq 
1:2, ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], 
length 1
14:28:43.537542 IP localhost.srvr > localhost.29105: Flags [F.], seq 2, 
ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537727 IP localhost.29105 > localhost.srvr: Flags [F.], seq 2, 
ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537741 IP localhost.srvr > localhost.29105: Flags [.], ack 3, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537895 IP localhost.29106 > localhost.srvr: Flags [S], seq 
1735077945:1735077946, win 43690, options [mss 65495,sackOK,TS val 
4294942098 ecr 0,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], 
length 1
14:28:43.537909 IP localhost.srvr > localhost.29106: Flags [S.], seq 
1983728126, ack 1735077947, win 43690, options [mss 65495,sackOK,TS val 
4294942098 ecr 4294942098,nop,wscale 7], length 0
14:28:43.537924 IP localhost.29106 > localhost.srvr: Flags [.], ack 1, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538014 IP localhost.srvr > localhost.29106: Flags [P.], seq 
1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], 
length 1
14:28:43.538028 IP localhost.srvr > localhost.29106: Flags [F.], seq 2, 
ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538196 IP localhost.29106 > localhost.srvr: Flags [.], ack 2, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538288 IP localhost.29106 > localhost.srvr: Flags [F.], seq 1, 
ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538300 IP localhost.srvr > localhost.29106: Flags [.], ack 2, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538417 IP localhost.29107 > localhost.srvr: Flags [S], seq 
3902541042:3902541043, win 43690, options [mss 65495,sackOK,TS val 
4294942098 ecr 0,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], 
length 1
14:28:43.538431 IP localhost.srvr > localhost.29107: Flags [S.], seq 
941945820, ack 3902541044, win 43690, options [mss 65495,sackOK,TS val 
4294942098 ecr 4294942098,nop,wscale 7], length 0
14:28:43.538445 IP localhost.29107 > localhost.srvr: Flags [.], ack 1, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538518 IP localhost.srvr > localhost.29107: Flags [P.], seq 
1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], 
length 1
14:28:43.538531 IP localhost.srvr > localhost.29107: Flags [F.], seq 2, 
ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538654 IP localhost.29107 > localhost.srvr: Flags [.], ack 2, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538730 IP localhost.29107 > localhost.srvr: Flags [F.], seq 1, 
ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538742 IP localhost.srvr > localhost.29107: Flags [.], ack 2, 
win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0

in which case I suppose that means that netperf top-of-trunk does indeed 
have client and server side support for TCP_FASTOPEN.  Enabled via the 
test-specific -F option though in the loopback test (in a 1 VCPU VM) I 
don't see much of a difference (don't suppose I should really):

raj@tardy-ubuntu-1204:~/netperf2_trunk/src$ ./netperf -t TCP_CRR -l 30 
-i 30,3 -- -F -P ,12345
MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 
AF_INET to localhost () port 12345 AF_INET : +/-2.500% @ 99% conf.  : demo
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       30.00    15909.07
16384  87380
raj@tardy-ubuntu-1204:~/netperf2_trunk/src$ ./netperf -t TCP_CRR -l 30 
-i 30,3 --  -P ,12345
MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 
AF_INET to localhost () port 12345 AF_INET : +/-2.500% @ 99% conf.  : demo
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       30.00    15574.37
16384  87380

happy benchmarking,

rick jones