TCP fast open question

TCP fast open question

[Zheng Liu]

Hi all,

Now we are trying to use TCP fast open in our nginx server, and we
encounter a problem under non-blocking socket.  I appreciate if some one
can reply this question.  Thanks in advance.

I describe our question here.  we have two machines, one is as server and
another is as client.  'net.ipv4.tcp_fastopen' on both of them are set to
3.  The server program looks like below:

  ... 
  listenfd = socket(AF_INET, SOCK_STREAM, 0);
  bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
  int tfo_opt = 1;
  setsockopt(listenfd, SOL_TCP, TCP_LISTEN_INFO, &tfo_opt, sizeof(tfo_opt));
  listen(listenfd, 5);
  connfd = accept(listenfd, (struct sockaddr *)NULL, NULL);
  recv(connfd, &buf, 4096)
  ...

The client program:

  ...
  sockfd = socket(AF_INET, SOCK_STREAM, 0);
  fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL, 0)|O_NONBLOCK);
  sendto(sockfd, msg, strlen(msg), MSG_FASTOPEN,
  	(struct sockaddr *)&servaddr, sizeof(servaddr);
  recv(sockfd, buf, 4096, 0);
  ...

We use a non-blocking socket to connect the server and send some
messages.  After calling sendto(2) we always get an EINPROGRESS error.
We think it is reasonable because connect(2) could also return this
error with a non-blocking socket and the connection will be established
later.  The question is *whether or not the data will be sent* after the
connection is established.  If I understand correctly, sendto(2) will
return the number of bytes of data queued up in the kernel or sent in
the SYN packet.  Even though the EINPROGRESS is returned.  If sendto(2)
returns -1, that means that no data is queued up in kernel or sent in
the packet.  Please correct me if I miss-understand something.

We run the program in our testing environment, and we use tcpdump(1) to
capture the packets.  From the result we can see there is no any data
that is sent.  Then we do another testing that after calling sendto(2)
the client program will sleep for 5 seconds and then call send(2) to
transfer some data.

  $ ./client xx.xx.xx.xx
  ret -1 errno 115

The program works well.  So that means that after getting a EINPROGRESS
error, the program must call send(2) manually.  Is it correct?  If we
want to avoid this problem, we need to set 'net.ipv4.tcp_fastopen' to
0x707 ?

Regards,
						- Zheng

Re: TCP fast open question

[Zhi Yong Wu]

On Sat, Mar 8, 2014 at 7:58 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> Hi all,
>
> Now we are trying to use TCP fast open in our nginx server, and we
> encounter a problem under non-blocking socket.  I appreciate if some one
> can reply this question.  Thanks in advance.
>
> I describe our question here.  we have two machines, one is as server and
> another is as client.  'net.ipv4.tcp_fastopen' on both of them are set to
> 3.  The server program looks like below:
>
>   ...
>   listenfd = socket(AF_INET, SOCK_STREAM, 0);
>   bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
>   int tfo_opt = 1;
>   setsockopt(listenfd, SOL_TCP, TCP_LISTEN_INFO, &tfo_opt, sizeof(tfo_opt));
s/TCP_LISTEN_INFO/TCP_FASTOPEN/??
>   listen(listenfd, 5);
>   connfd = accept(listenfd, (struct sockaddr *)NULL, NULL);
>   recv(connfd, &buf, 4096)
>   ...
>
> The client program:
>
>   ...
>   sockfd = socket(AF_INET, SOCK_STREAM, 0);
>   fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL, 0)|O_NONBLOCK);
>   sendto(sockfd, msg, strlen(msg), MSG_FASTOPEN,
>         (struct sockaddr *)&servaddr, sizeof(servaddr);
>   recv(sockfd, buf, 4096, 0);
>   ...
>
> We use a non-blocking socket to connect the server and send some
> messages.  After calling sendto(2) we always get an EINPROGRESS error.
> We think it is reasonable because connect(2) could also return this
> error with a non-blocking socket and the connection will be established
> later.  The question is *whether or not the data will be sent* after the
> connection is established.  If I understand correctly, sendto(2) will
> return the number of bytes of data queued up in the kernel or sent in
> the SYN packet.  Even though the EINPROGRESS is returned.  If sendto(2)
> returns -1, that means that no data is queued up in kernel or sent in
> the packet.  Please correct me if I miss-understand something.
>
> We run the program in our testing environment, and we use tcpdump(1) to
> capture the packets.  From the result we can see there is no any data
> that is sent.  Then we do another testing that after calling sendto(2)
> the client program will sleep for 5 seconds and then call send(2) to
> transfer some data.
>
>   $ ./client xx.xx.xx.xx
>   ret -1 errno 115
>
> The program works well.  So that means that after getting a EINPROGRESS
> error, the program must call send(2) manually.  Is it correct?  If we
> want to avoid this problem, we need to set 'net.ipv4.tcp_fastopen' to
> 0x707 ?
>
> Regards,
>                                                 - Zheng
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards,

Zhi Yong Wu

Re: TCP fast open question

[gnehzuil.liu]

> 在 2014年3月8日，下午8:56，Zhi Yong Wu <zwu.kernel@gmail.com> 写道：
> 
>> On Sat, Mar 8, 2014 at 7:58 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
>> Hi all,
>> 
>> Now we are trying to use TCP fast open in our nginx server, and we
>> encounter a problem under non-blocking socket.  I appreciate if some one
>> can reply this question.  Thanks in advance.
>> 
>> I describe our question here.  we have two machines, one is as server and
>> another is as client.  'net.ipv4.tcp_fastopen' on both of them are set to
>> 3.  The server program looks like below:
>> 
>>  ...
>>  listenfd = socket(AF_INET, SOCK_STREAM, 0);
>>  bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
>>  int tfo_opt = 1;
>>  setsockopt(listenfd, SOL_TCP, TCP_LISTEN_INFO, &tfo_opt, sizeof(tfo_opt));
> s/TCP_LISTEN_INFO/TCP_FASTOPEN/??

Sigh!  Yes, it's a typo.  Thanks for pointing it out.

Regards,
                                                - Zheng

>>  listen(listenfd, 5);
>>  connfd = accept(listenfd, (struct sockaddr *)NULL, NULL);
>>  recv(connfd, &buf, 4096)
>>  ...
>> 
>> The client program:
>> 
>>  ...
>>  sockfd = socket(AF_INET, SOCK_STREAM, 0);
>>  fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL, 0)|O_NONBLOCK);
>>  sendto(sockfd, msg, strlen(msg), MSG_FASTOPEN,
>>        (struct sockaddr *)&servaddr, sizeof(servaddr);
>>  recv(sockfd, buf, 4096, 0);
>>  ...
>> 
>> We use a non-blocking socket to connect the server and send some
>> messages.  After calling sendto(2) we always get an EINPROGRESS error.
>> We think it is reasonable because connect(2) could also return this
>> error with a non-blocking socket and the connection will be established
>> later.  The question is *whether or not the data will be sent* after the
>> connection is established.  If I understand correctly, sendto(2) will
>> return the number of bytes of data queued up in the kernel or sent in
>> the SYN packet.  Even though the EINPROGRESS is returned.  If sendto(2)
>> returns -1, that means that no data is queued up in kernel or sent in
>> the packet.  Please correct me if I miss-understand something.
>> 
>> We run the program in our testing environment, and we use tcpdump(1) to
>> capture the packets.  From the result we can see there is no any data
>> that is sent.  Then we do another testing that after calling sendto(2)
>> the client program will sleep for 5 seconds and then call send(2) to
>> transfer some data.
>> 
>>  $ ./client xx.xx.xx.xx
>>  ret -1 errno 115
>> 
>> The program works well.  So that means that after getting a EINPROGRESS
>> error, the program must call send(2) manually.  Is it correct?  If we
>> want to avoid this problem, we need to set 'net.ipv4.tcp_fastopen' to
>> 0x707 ?
>> 
>> Regards,
>>                                                - Zheng
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> Regards,
> 
> Zhi Yong Wu

Re: TCP fast open question

[Yuchung Cheng]

On Sat, Mar 8, 2014 at 3:58 AM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> Hi all,
>
> Now we are trying to use TCP fast open in our nginx server, and we
> encounter a problem under non-blocking socket.  I appreciate if some one
> can reply this question.  Thanks in advance.
>
> I describe our question here.  we have two machines, one is as server and
> another is as client.  'net.ipv4.tcp_fastopen' on both of them are set to
> 3.  The server program looks like below:
>
>   ...
>   listenfd = socket(AF_INET, SOCK_STREAM, 0);
>   bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
>   int tfo_opt = 1;
>   setsockopt(listenfd, SOL_TCP, TCP_LISTEN_INFO, &tfo_opt, sizeof(tfo_opt));
>   listen(listenfd, 5);
>   connfd = accept(listenfd, (struct sockaddr *)NULL, NULL);
>   recv(connfd, &buf, 4096)
>   ...
>
> The client program:
>
>   ...
>   sockfd = socket(AF_INET, SOCK_STREAM, 0);
>   fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL, 0)|O_NONBLOCK);
>   sendto(sockfd, msg, strlen(msg), MSG_FASTOPEN,
>         (struct sockaddr *)&servaddr, sizeof(servaddr);
>   recv(sockfd, buf, 4096, 0);
>   ...
>
> We use a non-blocking socket to connect the server and send some
> messages.  After calling sendto(2) we always get an EINPROGRESS error.
> We think it is reasonable because connect(2) could also return this
> error with a non-blocking socket and the connection will be established
> later.  The question is *whether or not the data will be sent* after the
> connection is established.  If I understand correctly, sendto(2) will
> return the number of bytes of data queued up in the kernel or sent in
> the SYN packet.  Even though the EINPROGRESS is returned.  If sendto(2)
> returns -1, that means that no data is queued up in kernel or sent in
> the packet.  Please correct me if I miss-understand something.
Correct.
http://tools.ietf.org/html/draft-ietf-tcpm-fastopen-07#appendix-A.1

>
> We run the program in our testing environment, and we use tcpdump(1) to
> capture the packets.  From the result we can see there is no any data
> that is sent.  Then we do another testing that after calling sendto(2)
The first handshake is used to get the Fast Open cookie for future
connections. I suspect you didn't enable Fast Open on the server side.
What're the output of 'sysctl net.ipv4.tcp_fastopen' on your client
and server? pls send the tcpdump traces on both sides as well.

> the client program will sleep for 5 seconds and then call send(2) to
> transfer some data.
>
>   $ ./client xx.xx.xx.xx
>   ret -1 errno 115
>
> The program works well.  So that means that after getting a EINPROGRESS
> error, the program must call send(2) manually.  Is it correct?  If we
> want to avoid this problem, we need to set 'net.ipv4.tcp_fastopen' to
> 0x707 ?
>
> Regards,
>                                                 - Zheng

Re: TCP fast open question

[Zheng Liu]

Hi Yuchung,

Sorry for the late reply because of an urgent issue.

On Mon, Mar 10, 2014 at 10:09:06AM -0700, Yuchung Cheng wrote:
> On Sat, Mar 8, 2014 at 3:58 AM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> > Hi all,
> >
> > Now we are trying to use TCP fast open in our nginx server, and we
> > encounter a problem under non-blocking socket.  I appreciate if some one
> > can reply this question.  Thanks in advance.
> >
> > I describe our question here.  we have two machines, one is as server and
> > another is as client.  'net.ipv4.tcp_fastopen' on both of them are set to
> > 3.  The server program looks like below:
> >
> >   ...
> >   listenfd = socket(AF_INET, SOCK_STREAM, 0);
> >   bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
> >   int tfo_opt = 1;
> >   setsockopt(listenfd, SOL_TCP, TCP_LISTEN_INFO, &tfo_opt, sizeof(tfo_opt));
> >   listen(listenfd, 5);
> >   connfd = accept(listenfd, (struct sockaddr *)NULL, NULL);
> >   recv(connfd, &buf, 4096)
> >   ...
> >
> > The client program:
> >
> >   ...
> >   sockfd = socket(AF_INET, SOCK_STREAM, 0);
> >   fcntl(sockfd, F_SETFL, fcntl(sockfd, F_GETFL, 0)|O_NONBLOCK);
> >   sendto(sockfd, msg, strlen(msg), MSG_FASTOPEN,
> >         (struct sockaddr *)&servaddr, sizeof(servaddr);
> >   recv(sockfd, buf, 4096, 0);
> >   ...
> >
> > We use a non-blocking socket to connect the server and send some
> > messages.  After calling sendto(2) we always get an EINPROGRESS error.
> > We think it is reasonable because connect(2) could also return this
> > error with a non-blocking socket and the connection will be established
> > later.  The question is *whether or not the data will be sent* after the
> > connection is established.  If I understand correctly, sendto(2) will
> > return the number of bytes of data queued up in the kernel or sent in
> > the SYN packet.  Even though the EINPROGRESS is returned.  If sendto(2)
> > returns -1, that means that no data is queued up in kernel or sent in
> > the packet.  Please correct me if I miss-understand something.
> Correct.
> http://tools.ietf.org/html/draft-ietf-tcpm-fastopen-07#appendix-A.1
> 
> >
> > We run the program in our testing environment, and we use tcpdump(1) to
> > capture the packets.  From the result we can see there is no any data
> > that is sent.  Then we do another testing that after calling sendto(2)
> The first handshake is used to get the Fast Open cookie for future
> connections. I suspect you didn't enable Fast Open on the server side.
> What're the output of 'sysctl net.ipv4.tcp_fastopen' on your client
> and server? pls send the tcpdump traces on both sides as well.

Sorry, let me clarify the description please.  We can capture the syn
packet without data.  So that means that client has issued a cookie
request.  My question is when the data could be sent out.

Here I miss understand the behaviour when the program is the first time
to call sendto(2).  As RFC draft of TFO described, sendto(2) will return
-1 with EINPROGRESS error if the cookie is not available locallly.  That
means that we need to call write(2) to send the data after the connection
is established.  Namely, after getting -1 with EINPROGRESS, the caller
must call write(2) again later.  Now this problem has been solved.

Thanks for your help,
                                                - Zheng