socket write(2) after remote shutdown(2) problem ?

socket write(2) after remote shutdown(2) problem ?

[Stelian Pop]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

Hi,

Is the following behaviour correct on a tcp connection: 
	* the server issues a shutdown(sock, RW)
	* the client side socket passes in CLOSE-WAIT state
	* the client issues a write on the socket which succeds.

I expected the last write to fail, since the other side is not
capable any more to receive data. Can someone confirm to me one
of the following:
	1. behaviour is correct and why.
	2. shutdown is buggy.
	3. write is buggy.

Thanks.

Attached are sample codes for the "server" and the "client". Test
was done on latest 2.4 and 2.5 kernels.

Stelian.
-- 
Stelian Pop <stelian.pop@fr.alcove.com>
Alcove - http://www.alcove.com

[-- Attachment #2: client.c --]
[-- Type: text/plain, Size: 1931 bytes --]

#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/select.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <signal.h>

char buf[1024];

int main()
{
  int fd;
  char * ptr;
  struct sockaddr_in addr;
  int ready = 0;
  fd_set rfds, efds;
  int nread;

  /* establish connection */

  fd = socket (PF_INET, SOCK_STREAM, 0);
  if ( -1 == fd) {
    perror ("socket");
    exit (1);
  }

  addr.sin_family = AF_INET;
  addr.sin_addr.s_addr = htonl (INADDR_ANY);
  addr.sin_port = 0;
  if (bind (fd, (struct sockaddr*)&addr, sizeof (struct sockaddr_in)) != 0) {
    perror ("bind");
    exit (1);
  }

  ptr = (char*)&addr.sin_addr.s_addr;
  /*  ptr[0] = 162; ptr[1] = 0; ptr[2] = 120; ptr[3] = 98;*/
  ptr[0] = 127; ptr[1] = 0; ptr[2] = 0; ptr[3] = 1;
  addr.sin_port = htons(9999);
  if (connect (fd, (struct sockaddr*)&addr, sizeof(addr)) != 0) {
    perror ("connect");
    exit (1);
  }
  
  FD_ZERO (&rfds);
  FD_SET (fd, &rfds);
  select (fd+1, &rfds, 0, 0, 0);

  /* read everything from socket and print it to stdout */

  while (!ready) {
    struct timeval tv;

    FD_ZERO (&rfds);
    FD_SET (fd, &rfds);
    FD_ZERO (&efds);
    FD_SET (fd, &efds);
    memset (&tv, 0, sizeof(tv));

    switch (select (fd+1, &rfds, 0, &efds, &tv)) {
    case -1:
      perror("select");
      exit (1);
    case 0:
      ready = 1;
      break;
    case 1:
      if (FD_ISSET (fd, &efds)) {
	fprintf (stderr, "select found exception");
      }
      nread = read (fd, &buf, 1024);
      if (nread == 0)
	ready = 1;
      else
	write (1, &buf, nread);
      break;
    }
  }

  /* here lsof will show CLOSE_WAIT */
  raise (SIGSTOP);

{
int ret;

  if ((ret = write (fd, "QUIT\n", 5)) == -1) {
    perror ("write");
    exit (1);
  }
  printf("write ok, ret=%d\n", ret);
}

  /* here the socket is orphaned, after write succeeded */
  raise (SIGSTOP);

  return 0;
}

[-- Attachment #3: server.c --]
[-- Type: text/plain, Size: 1071 bytes --]

#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/select.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <signal.h>

char buf[1024];

int main()
{
  int fd, conn;
  struct sockaddr_in addr;
  socklen_t size;

  /* get socket to client, listening on 9999 */

  fd = socket (PF_INET, SOCK_STREAM, 0);
  if ( -1 == fd) {
    perror ("socket");
    exit (1);
  }

  addr.sin_family = AF_INET;
  addr.sin_addr.s_addr = htonl (INADDR_ANY);
  addr.sin_port = htons(9999);
  if (bind (fd, (struct sockaddr*)&addr, sizeof (struct sockaddr_in)) != 0) {
    perror ("bind");
    exit (1);
  }

  if (-1 == listen (fd, 1)) {
    perror ("listen");
    exit (1);
  }

  if (-1 == (conn = accept(fd, (struct sockaddr*)&addr, &size))) {
    perror ("accept");
    exit (1);
  }

  /* send some info */

  if (write (conn, "Hello client !\n", 15) == -1) {
    perror ("write");
    exit (1);
  }

  /* finish */

  if (-1 == shutdown (conn, SHUT_RDWR)) {
    perror ("shutdown");
    exit (1);
  }

  return 0;
}

Re: socket write(2) after remote shutdown(2) problem ?

[David S. Miller]

   From: Stelian Pop <stelian.pop@fr.alcove.com>
   Date: Fri, 5 Apr 2002 11:50:39 +0200

   	* the client side socket passes in CLOSE-WAIT state
   	* the client issues a write on the socket which succeds.
 ...   
   Attached are sample codes for the "server" and the "client". Test
   was done on latest 2.4 and 2.5 kernels.

Your client does not do any write()'s after the shutdown call.
It simply exit(0)'s.

Re: socket write(2) after remote shutdown(2) problem ?

[David S. Miller]

   From: Stelian Pop <stelian.pop@fr.alcove.com>
   Date: Fri, 5 Apr 2002 12:47:33 +0200

   On Fri, Apr 05, 2002 at 02:04:43AM -0800, David S. Miller wrote:
   
   > Your client does not do any write()'s after the shutdown call.
   > It simply exit(0)'s.
   
   You mean the 'server' ? Even if I add a sleep(600) between the 
   shutdown() call and the exit() call I get the same behaviour.

Oh I see now.  Here is what should happen:

	* server shutdown(ALL)
	* the write() should succeed on the client
	* client socket receives a TCP reset

If this isn't what is happening, send us a trace.

Re: socket write(2) after remote shutdown(2) problem ?

[Stelian Pop]

On Fri, Apr 05, 2002 at 02:04:43AM -0800, David S. Miller wrote:

>    From: Stelian Pop <stelian.pop@fr.alcove.com>
>    Date: Fri, 5 Apr 2002 11:50:39 +0200
> 
>    	* the client side socket passes in CLOSE-WAIT state
>    	* the client issues a write on the socket which succeds.
>  ...   
>    Attached are sample codes for the "server" and the "client". Test
>    was done on latest 2.4 and 2.5 kernels.
> 
> Your client does not do any write()'s after the shutdown call.
> It simply exit(0)'s.

You mean the 'server' ? Even if I add a sleep(600) between the 
shutdown() call and the exit() call I get the same behaviour.

Stelian.
-- 
Stelian Pop <stelian.pop@fr.alcove.com>
Alcove - http://www.alcove.com

Re: socket write(2) after remote shutdown(2) problem ?

[Stelian Pop]

On Fri, Apr 05, 2002 at 02:44:35AM -0800, David S. Miller wrote:

>    > Your client does not do any write()'s after the shutdown call.
>    > It simply exit(0)'s.
>    
>    You mean the 'server' ? Even if I add a sleep(600) between the 
>    shutdown() call and the exit() call I get the same behaviour.
> 
> Oh I see now.  Here is what should happen:
> 
> 	* server shutdown(ALL)
> 	* the write() should succeed on the client

So this is really supposed to succeed then... Somewhat illogical to
me...

> 	* client socket receives a TCP reset

How is the client socket supposed to know it received a TCP reset
(I am talking from the application point of view, not the kernel...) ?

Stelian.
-- 
Stelian Pop <stelian.pop@fr.alcove.com>
Alcove - http://www.alcove.com

Re: socket write(2) after remote shutdown(2) problem ?

[David S. Miller]

   From: Stelian Pop <stelian.pop@fr.alcove.com>
   Date: Fri, 5 Apr 2002 12:55:09 +0200
   
   > 	* client socket receives a TCP reset
   
   How is the client socket supposed to know it received a TCP reset
   (I am talking from the application point of view, not the kernel...) ?

You may find out by attempting to read data, or you may use the
extended IP error reporting Linux has.

But all of this is irrelevant.  When a server closes and says "send me
no more data", this implies that the server told the client it doesn't
want any more data.  If the client sends data, this is a gross fatal
error, so TCP resets in FIN_WAIT{1,2} states.

RFC 793 originally specified to queue the data, RFC 1122 is where
the current behavior is defined.

Re: socket write(2) after remote shutdown(2) problem ?

[Andi Kleen]

"David S. Miller" <davem@redhat.com> writes:
> 
> But all of this is irrelevant.  When a server closes and says "send me
> no more data", this implies that the server told the client it doesn't
> want any more data.  If the client sends data, this is a gross fatal
> error, so TCP resets in FIN_WAIT{1,2} states.

Linux has one hole in this. When the reset arrives too late the pending
error is not passed up and propagated through shutdown/close.

-Andi

Re: socket write(2) after remote shutdown(2) problem ?

[Stelian Pop]

On Fri, Apr 05, 2002 at 03:02:51AM -0800, David S. Miller wrote:

>    From: Stelian Pop <stelian.pop@fr.alcove.com>
>    Date: Fri, 5 Apr 2002 12:55:09 +0200
>    
>    > 	* client socket receives a TCP reset
>    
>    How is the client socket supposed to know it received a TCP reset
>    (I am talking from the application point of view, not the kernel...) ?
> 
> You may find out by attempting to read data, or you may use the
> extended IP error reporting Linux has.

In the client, I've added a read(fd, buf, 1) just after the write,
and rerun:

	client				server

	...				...
					shutdown()
	...		
lsof will shown the client socket being in CLOSE_WAIT at this point
					...
	write(fd, buf, 5) = 5
	read(fd, buf, 1) = 0
	...
	exit(0)
					exit(0)

As you can see, read() doesn't return any error, just 0 to 
indicate end-of-file (seems correct interpretation of remote
shutdown here), but it doesn't report any error from the 
precedent write... Bug ?

> But all of this is irrelevant.  When a server closes and says "send me
> no more data", this implies that the server told the client it doesn't
> want any more data.

Perfectly valid but only if the 'applicative close' has reached the
other end. If it's still queued in TCP buffers, the client may not
have received yet that close... I thought that this was the only
reason that shutdown() existed at all: flush the buffers preparing
an imminent close...

> If the client sends data, this is a gross fatal
> error, so TCP resets in FIN_WAIT{1,2} states.
> 
> RFC 793 originally specified to queue the data, RFC 1122 is where
> the current behavior is defined.

Thanks for the pointer. 

Stelian.
-- 
Stelian Pop <stelian.pop@fr.alcove.com>
Alcove - http://www.alcove.com

Re: socket write(2) after remote shutdown(2) problem ?

[David S. Miller]

   From: Stelian Pop <stelian.pop@fr.alcove.com>
   Date: Fri, 5 Apr 2002 14:00:55 +0200

   As you can see, read() doesn't return any error, just 0 to 
   indicate end-of-file (seems correct interpretation of remote
   shutdown here), but it doesn't report any error from the 
   precedent write... Bug ?

Race, wait a bit, the reset will arrive.

This is a error state in your code, writing when the server expects no
data.  Always remember that when you are trying to judge if
whatever socket operation results is valid or invalid.
   
Since we find out about the reset asynchronously, this is what
happens.

   > But all of this is irrelevant.  When a server closes and says "send me
   > no more data", this implies that the server told the client it doesn't
   > want any more data.
   
   Perfectly valid but only if the 'applicative close' has reached the
   other end. If it's still queued in TCP buffers, the client may not
   have received yet that close... I thought that this was the only
   reason that shutdown() existed at all: flush the buffers preparing
   an imminent close...

Not a close "in TCP" but a close "in your apps protocol".
As in:

	write(socket_fd, "Ok client I have all your data, lets close now")

See?  If the server closes without telling the client that, all
bets are off.

Look, your app is buggy, PERIOD.  Once you start to write to a closed
socket, sorry the phase of the moon decides what happens to you.  Most
of the time you'll be lucky and see an error.

Re: socket write(2) after remote shutdown(2) problem ?

[Stelian Pop]

On Fri, Apr 05, 2002 at 04:04:51AM -0800, David S. Miller wrote:

>    From: Stelian Pop <stelian.pop@fr.alcove.com>
>    Date: Fri, 5 Apr 2002 14:00:55 +0200
> 
>    As you can see, read() doesn't return any error, just 0 to 
>    indicate end-of-file (seems correct interpretation of remote
>    shutdown here), but it doesn't report any error from the 
>    precedent write... Bug ?
> 
> Race, wait a bit, the reset will arrive.

30 seconds later the read still returns 0...

> Look, your app is buggy, PERIOD.  Once you start to write to a closed
> socket, sorry the phase of the moon decides what happens to you.  Most
> of the time you'll be lucky and see an error.

The socket is not closed on the client side. I expect the kernel to
signal me an error (reset of read/write return code, whatever) if
the connection is closed by the _server_ (with close OR shutdown).

Stelian.
-- 
Stelian Pop <stelian.pop@fr.alcove.com>
Alcove - http://www.alcove.com

Re: socket write(2) after remote shutdown(2) problem ?

[Stelian Pop]

On Fri, Apr 05, 2002 at 04:04:51AM -0800, David S. Miller wrote:

>    As you can see, read() doesn't return any error, just 0 to 
>    indicate end-of-file (seems correct interpretation of remote
>    shutdown here), but it doesn't report any error from the 
>    precedent write... Bug ?
> 
> Race, wait a bit, the reset will arrive.

Ok, I investigated this a bit more using setsockopt(...,SO_ERROR,...)

After the write in the client (which is done after the server has
shutdown()'ed it), the error bit is set on the client socket
(-EPIPE).

If the client issues a second write, the write fails (correctly)
setting errno to -EPIPE.

If the client issues a read, read doesn't return an error. The 
socket error bit is still there however, even after read() returns.

If the client issues a close, close will return 0 too.

Whether the read() should return the error bit or not is
debatable, but IMHO close at least should propagate the error.

Stelian.
-- 
Stelian Pop <stelian.pop@fr.alcove.com>
Alcove - http://www.alcove.com