SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

[Tomas Hozza]

[-- Attachment #1: Type: text/plain, Size: 2133 bytes --]

Hi.

I was advised to forward my networking Kernel related Bug here
so it can get fixed much faster. Here is the Bug description
copied from Red Hat Bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=912382):

Description of problem:
I'm using SOCK_STREAM TCP sockets. Imagine server-client application.
- Server will accept connection from client.
- Client sends some data to server -> server receives data.
- Client then half-closes the TCP connection using shutdown(fd, SHUT_WR).
- Client waits for some data from the server.
- Server sends some data to client -> client receives data.
- Client exits and closes the socket.
- After some time server tries to send some data to the client.
- The first send() call returns success even the Client's response is RST.


Version-Release number of selected component (if applicable):
3.7.7-201.fc18.x86_64
3.7.8-202.fc18.x86_64


How reproducible:
always


Steps to Reproduce:
1. download attached client/server sources.
2. compile server: # gcc tcp_server.c -o server
3. compile client: # gcc tcp_client.c -o client
4. run the server: # ./server
5. run the client: # ./client


Actual results:
Server started...
Created SOCK_STREAM socket
Bound to localhost:6666
waiting for connection...
Client connected!
Waiting for some data from client
Received: "MSG from client"
Received EOF
Sending msg to client: "MSG from server"
sleep(5)
Sending 1 msg to client: "MSG from server"
try #1 send() result: (0) Success
Sending 2 msg to client: "MSG from server"
try #2 send() result: (32) Broken pipe
Server exiting...


Expected results:
Server should fail to send() data right after sleep().

Server started...
Created SOCK_STREAM socket
Bound to localhost:6666
waiting for connection...
Client connected!
Waiting for some data from client
Received: "MSG from client"
Received EOF
Sending msg to client: "MSG from server"
sleep(5)
Sending 1 msg to client: "MSG from server"
try #1 send() result: (32) Broken pipe
Server exiting...


Additional info:
RST answer means that data were NOT delivered. Therefore the very
first send() call after server sleep() should fail!


Regards,

Tomas Hozza

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tcp_server.c --]
[-- Type: text/x-csrc; name=tcp_server.c, Size: 2474 bytes --]

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <signal.h>

int main(int argc, char **argv)
{
    int ssock, csock, i;
    static char buff[256];
    ssize_t nbytes;
    struct sockaddr_in saddr;

    memset(&saddr, 0, sizeof(saddr));
    saddr.sin_family = AF_INET;
    saddr.sin_port = htons(6666);

    signal(SIGPIPE, SIG_IGN);

    printf("Server started...\n");

    if ((ssock = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
        printf("socket() error. (%d) %s\n", errno, strerror(errno));
        exit(1);
    }
    printf("Created SOCK_STREAM socket\n");

    if (bind(ssock, (struct sockaddr*)&saddr, sizeof(saddr)) == -1) {
        printf("bind() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }
    printf("Bound to localhost:6666\n");

    if (listen(ssock, 1) == -1) {
        printf("listen() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }

    printf("waiting for connection...\n");
    csock = accept(ssock, NULL, NULL);

    if (csock == -1) {
        printf("accept() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }
    printf("Client connected!\n");
    printf("Waiting for some data from client\n");

    while ((nbytes = recv(csock, buff, sizeof(buff), 0)) > 0) {
            printf("Received: \"%s\"\n", buff);
    }

    if (nbytes < 0) {
        printf("recv() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        close(csock);
        exit(1);
    } else if (nbytes == 0) {
        printf("Received EOF\n");
        printf("Sending msg to client: \"MSG from server\"\n");
        if ((nbytes = send(csock, "MSG from server", sizeof("MSG from server"), 0)) <= 0) {
            printf("send() error. (%d) %s\n", errno, strerror(errno));
            close(ssock);
            close(csock);
            exit(1);
        }
    }

    printf("sleep(5)\n");
    sleep(5);
    
    i = 1;
    while (1) { 
        printf("Sending %d msg to client: \"MSG from server\"\n", i);
        nbytes = send(csock, "MSG from server", sizeof("MSG from server"), 0);
        printf("try #%d send() result: (%d) %s\n",i ,errno, strerror(errno));
        if (nbytes <= 0)
            break;
        ++i;
    }

    printf("Server exiting...\n");
    close(ssock);
    close(csock);
    return 0;
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: tcp_client.c --]
[-- Type: text/x-csrc; name=tcp_client.c, Size: 1694 bytes --]

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

int main(int argc, char **argv)
{
    int ssock, i;
    static char buff[256];
    ssize_t nbytes;
    struct sockaddr_in saddr;

    memset(&saddr, 0, sizeof(saddr));
    saddr.sin_family = AF_INET;
    saddr.sin_port = htons(6666);

    printf("Client started...\n");

    if ((ssock = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
        printf("socket() error. (%d) %s\n", errno, strerror(errno));
        exit(1);
    }

    printf("Created SOCK_STREAM socket\n");

    if (connect(ssock, (struct sockaddr *)&saddr, sizeof(saddr)) == -1) {
        printf("connect() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }

    printf("Connected to the server!\n");
    printf("Sending some data to server\n");

    if ((nbytes = send(ssock, "MSG from client", sizeof("MSG from client"), 0)) <= 0) {
        printf("send() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }

    printf("Half-closing connection with shutdown(ssock, SHUT_WR)\n");

    if (shutdown(ssock, SHUT_WR) == -1) {
        printf("shutdown() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }

    printf("Receiving data from server\n");
    if ((nbytes = recv(ssock, buff, sizeof(buff), 0)) > 0) {
            printf("Received: \"%s\"\n", buff);
    } else {
        printf("recv() error. (%d) %s\n", errno, strerror(errno));
        close(ssock);
        exit(1);
    }

    printf("Client exiting...\n");
    close(ssock);
    return 0;
}

Re: SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

[Eric Dumazet]

On Tue, 2013-02-19 at 02:02 -0500, Tomas Hozza wrote:
> Hi.
> 
> I was advised to forward my networking Kernel related Bug here
> so it can get fixed much faster. Here is the Bug description
> copied from Red Hat Bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=912382):
> 
> Description of problem:
> I'm using SOCK_STREAM TCP sockets. Imagine server-client application.
> - Server will accept connection from client.
> - Client sends some data to server -> server receives data.
> - Client then half-closes the TCP connection using shutdown(fd, SHUT_WR).
> - Client waits for some data from the server.
> - Server sends some data to client -> client receives data.
> - Client exits and closes the socket.
> - After some time server tries to send some data to the client.
> - The first send() call returns success even the Client's response is RST.
> 
> 
> Version-Release number of selected component (if applicable):
> 3.7.7-201.fc18.x86_64
> 3.7.8-202.fc18.x86_64
> 
> 
> How reproducible:
> always
> 
> 
> Steps to Reproduce:
> 1. download attached client/server sources.
> 2. compile server: # gcc tcp_server.c -o server
> 3. compile client: # gcc tcp_client.c -o client
> 4. run the server: # ./server
> 5. run the client: # ./client
> 
> 
> Actual results:
> Server started...
> Created SOCK_STREAM socket
> Bound to localhost:6666
> waiting for connection...
> Client connected!
> Waiting for some data from client
> Received: "MSG from client"
> Received EOF
> Sending msg to client: "MSG from server"
> sleep(5)
> Sending 1 msg to client: "MSG from server"
> try #1 send() result: (0) Success
> Sending 2 msg to client: "MSG from server"
> try #2 send() result: (32) Broken pipe
> Server exiting...
> 
> 
> Expected results:
> Server should fail to send() data right after sleep().
> 
> Server started...
> Created SOCK_STREAM socket
> Bound to localhost:6666
> waiting for connection...
> Client connected!
> Waiting for some data from client
> Received: "MSG from client"
> Received EOF
> Sending msg to client: "MSG from server"
> sleep(5)
> Sending 1 msg to client: "MSG from server"
> try #1 send() result: (32) Broken pipe
> Server exiting...
> 
> 
> Additional info:
> RST answer means that data were NOT delivered. Therefore the very
> first send() call after server sleep() should fail!
> 
> 
> Regards,
> 
> Tomas Hozza

Hi Tomas

I read your bug report, but cant understand why you believe there is a
bug.

send() is not synchronous.

It only queues bytes to be transmitted either right now, either later
when its possible.

So a RST message can be received _after_ send() has accepted the request
and copy data from user land to kernel socket write queue, but TCP stack
did not yet delivered the data.

To make sure data is delivered, you need additional logic.

Re: SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

[Tomas Hozza]

----- Original Message -----
> On Tue, 2013-02-19 at 02:02 -0500, Tomas Hozza wrote:
> > Hi.
> > 
> > I was advised to forward my networking Kernel related Bug here
> > so it can get fixed much faster. Here is the Bug description
> > copied from Red Hat Bugzilla
> > (https://bugzilla.redhat.com/show_bug.cgi?id=912382):
> > 
> > Description of problem:
> > I'm using SOCK_STREAM TCP sockets. Imagine server-client
> > application.
> > - Server will accept connection from client.
> > - Client sends some data to server -> server receives data.
> > - Client then half-closes the TCP connection using shutdown(fd,
> > SHUT_WR).
> > - Client waits for some data from the server.
> > - Server sends some data to client -> client receives data.
> > - Client exits and closes the socket.
> > - After some time server tries to send some data to the client.
> > - The first send() call returns success even the Client's response
> > is RST.
> > 
> > 
> > Version-Release number of selected component (if applicable):
> > 3.7.7-201.fc18.x86_64
> > 3.7.8-202.fc18.x86_64
> > 
> > 
> > How reproducible:
> > always
> > 
> > 
> > Steps to Reproduce:
> > 1. download attached client/server sources.
> > 2. compile server: # gcc tcp_server.c -o server
> > 3. compile client: # gcc tcp_client.c -o client
> > 4. run the server: # ./server
> > 5. run the client: # ./client
> > 
> > 
> > Actual results:
> > Server started...
> > Created SOCK_STREAM socket
> > Bound to localhost:6666
> > waiting for connection...
> > Client connected!
> > Waiting for some data from client
> > Received: "MSG from client"
> > Received EOF
> > Sending msg to client: "MSG from server"
> > sleep(5)
> > Sending 1 msg to client: "MSG from server"
> > try #1 send() result: (0) Success
> > Sending 2 msg to client: "MSG from server"
> > try #2 send() result: (32) Broken pipe
> > Server exiting...
> > 
> > 
> > Expected results:
> > Server should fail to send() data right after sleep().
> > 
> > Server started...
> > Created SOCK_STREAM socket
> > Bound to localhost:6666
> > waiting for connection...
> > Client connected!
> > Waiting for some data from client
> > Received: "MSG from client"
> > Received EOF
> > Sending msg to client: "MSG from server"
> > sleep(5)
> > Sending 1 msg to client: "MSG from server"
> > try #1 send() result: (32) Broken pipe
> > Server exiting...
> > 
> > 
> > Additional info:
> > RST answer means that data were NOT delivered. Therefore the very
> > first send() call after server sleep() should fail!
> > 
> > 
> > Regards,
> > 
> > Tomas Hozza
> 
> Hi Tomas
> 
> I read your bug report, but cant understand why you believe there is
> a bug.
> 
> send() is not synchronous.

OK, my bad. In man page there is: "Successful completion of a call to
send() does not guarantee delivery of the message."

> 
> It only queues bytes to be transmitted either right now, either later
> when its possible.
> 
> So a RST message can be received _after_ send() has accepted the
> request and copy data from user land to kernel socket write queue, but TCP
> stack did not yet delivered the data.

I'm using TCP because it should guarantee that my data were delivered or
let me know there was some problem. If this is not a bug, then it is at least
confusing for TCP.

> To make sure data is delivered, you need additional logic.

To be honest I didn't find any way how to get notified there was a RST packet
sent as a reply to my previously sent data.

Thank you for your reply.

Regards,
Tomas Hozza

Re: SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

[Stephen Hemminger]

> stack did not yet delivered the data.
> 
> I'm using TCP because it should guarantee that my data were delivered or
> let me know there was some problem. If this is not a bug, then it is at least
> confusing for TCP.
> 
> > To make sure data is delivered, you need additional logic.
> 
> To be honest I didn't find any way how to get notified there was a RST packet
> sent as a reply to my previously sent data.

TCP makes no end-to-end (application to application) guarantees.
Applications that need assurance need to do a request-reply at end of
transaction. You should get a good textbook like Stevens for more detail

Re: SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

[Eric Dumazet]

On Tue, 2013-02-19 at 10:09 -0500, Tomas Hozza wrote:

> I'm using TCP because it should guarantee that my data were delivered or
> let me know there was some problem. If this is not a bug, then it is at least
> confusing for TCP.

Note that a write() on a regular file descriptor has same semantic :
By default, there is no guarantee data is written on stable storage.

> 
> > To make sure data is delivered, you need additional logic.
> 
> To be honest I didn't find any way how to get notified there was a RST packet
> sent as a reply to my previously sent data.

Well, I suggest you read the man pages and some books, as this is well
explained, you are not the first guy wanting to exchange data using TCP.

man 7 socket

       SO_LINGER
              Sets or gets the SO_LINGER option.  The argument is a linger structure.

                  struct linger {
                      int l_onoff;    /* linger active */
                      int l_linger;   /* how many seconds to linger for */
                  };

              When enabled, a close(2) or shutdown(2) will not return until all queued messages
              for  the  socket  have  been  successfully  sent  or  the linger timeout has been
              reached.  Otherwise, the call returns immediately and the closing is done in  the
              background.   When  the socket is closed as part of exit(2), it always lingers in
              the background.

man 2 shutdown
...

Re: SOCK_STREAM TCP: send() returns success even when other side responded with RST packet

[Tomas Hozza]

----- Original Message -----
> On Tue, 2013-02-19 at 10:09 -0500, Tomas Hozza wrote:
> 
> > I'm using TCP because it should guarantee that my data were
> > delivered or
> > let me know there was some problem. If this is not a bug, then it
> > is at least
> > confusing for TCP.
> 
> Note that a write() on a regular file descriptor has same semantic :
> By default, there is no guarantee data is written on stable storage.
> 
> > 
> > > To make sure data is delivered, you need additional logic.
> > 
> > To be honest I didn't find any way how to get notified there was a
> > RST packet
> > sent as a reply to my previously sent data.
> 
> Well, I suggest you read the man pages and some books, as this is
> well
> explained, you are not the first guy wanting to exchange data using
> TCP.
> 
> man 7 socket
> 
>        SO_LINGER
>               Sets or gets the SO_LINGER option.  The argument is a
>               linger structure.
> 
>                   struct linger {
>                       int l_onoff;    /* linger active */
>                       int l_linger;   /* how many seconds to linger
>                       for */
>                   };
> 
>               When enabled, a close(2) or shutdown(2) will not return
>               until all queued messages
>               for  the  socket  have  been  successfully  sent  or
>                the linger timeout has been
>               reached.  Otherwise, the call returns immediately and
>               the closing is done in  the
>               background.   When  the socket is closed as part of
>               exit(2), it always lingers in
>               the background.
> 
> man 2 shutdown

I don't think you understood what I was asking for and this is not the right
place to discuss how to do things. I have read Unix Network Programming from
Richard Stevens but did not find answer for this either. Thanks anyway.