why would EPIPE cause socket port to change?

why would EPIPE cause socket port to change?

[dean gaudet]

in the test program below the getsockname result on a TCP socket changes 
across a write which produces EPIPE... here's a fragment of the strace:

getsockname(3, {sa_family=AF_INET, sin_port=htons(37636), sin_addr=inet_addr("127.0.0.1")}, [17863593746633850896]) = 0
...
write(3, "hi!\n", 4)                    = 4
write(3, "hi!\n", 4)                    = -1 EPIPE (Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
getsockname(3, {sa_family=AF_INET, sin_port=htons(59882), sin_addr=inet_addr("127.0.0.1")}, [16927060683038654480]) = 0

why does the port# change?  this is on 2.6.19.1.

(fwiw this is one of two reasons i've found for libnss-ldap to leak 
sockets... causing nscd to crash.)

-dean

reproduce like:

make test-sockname-change
nc -l -p 9999 -c "exit 0" &
strace ./test-sockname-change 127.0.0.1 9999

--- snip ---

#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <stdlib.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/uio.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>

#ifndef INADDR_NONE
#define INADDR_NONE (-1ul)
#endif

int main(int argc, char **argv)
{
  struct sockaddr_in server_addr;
  struct sockaddr_in before, after;
  socklen_t slen;
  int s;
  struct iovec vector[3];
  char buf[100];
  int i;
  const int just_say_no = 1;

  if (argc != 3) {
usage:
    fprintf(stderr, "usage: test-sigpipe a.b.c.d port#\n");
    exit(1);
  }
  server_addr.sin_family = AF_INET;
  server_addr.sin_addr.s_addr = inet_addr(argv[1]);
  if (server_addr.sin_addr.s_addr == INADDR_NONE) {
    fprintf(stderr, "bogus address\n");
    goto usage;
  }
  server_addr.sin_port = htons(atoi(argv[2]));

  s = socket(AF_INET, SOCK_STREAM, 0);
  if (s < 0) {
    perror("socket");
    exit(1);
  }
  if (connect(s, (struct sockaddr *)&server_addr, sizeof(server_addr)) != 0) {
    perror("connect");
    exit(1);
  }

  if (setsockopt(s, IPPROTO_TCP, TCP_NODELAY, (char*)&just_say_no, sizeof(just_say_no)) != 0) {
    perror( "TCP_NODELAY" );
    exit(1);
  }

  fcntl(s, F_SETFL, fcntl(s, F_GETFL) | O_NONBLOCK);

  slen = sizeof(before);
  if (getsockname(s, (struct sockaddr *)&before, &slen)) {
    perror("getsockname before");
  }

  signal(SIGPIPE, SIG_IGN);

  sleep(1);

  do {
    i = write(s, "hi!\n", 4);
  } while (i >= 0);
  if (errno != EPIPE) {
    fprintf(stderr, "was expecting EPIPE from write\n");
    exit(1);
  }

  slen = sizeof(after);
  if (getsockname(s, (struct sockaddr *)&after, &slen)) {
    perror("getsockname after");
  }

  printf("before = %d, after = %d\n", ntohs(before.sin_port), ntohs(after.sin_port));

  return 0;
}

Re: why would EPIPE cause socket port to change?

[Herbert Xu]

dean gaudet <dean@arctic.org> wrote:
> in the test program below the getsockname result on a TCP socket changes 
> across a write which produces EPIPE... here's a fragment of the strace:
> 
> getsockname(3, {sa_family=AF_INET, sin_port=htons(37636), sin_addr=inet_addr("127.0.0.1")}, [17863593746633850896]) = 0
> ...
> write(3, "hi!\n", 4)                    = 4
> write(3, "hi!\n", 4)                    = -1 EPIPE (Broken pipe)
> --- SIGPIPE (Broken pipe) @ 0 (0) ---
> getsockname(3, {sa_family=AF_INET, sin_port=htons(59882), sin_addr=inet_addr("127.0.0.1")}, [16927060683038654480]) = 0
> 
> why does the port# change?  this is on 2.6.19.1.

Prior to the last write, the socket entered the CLOSED state meaning
that the old port is no longer allocated to it.  As a result, the
last write operates on an unconnected socket which causes a new local
port to be allocated as an autobind.  It then fails because the socket
is still not connected.

So any attempt to run getsockname after an error on the socket is
simply buggy.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: why would EPIPE cause socket port to change?

[Michael Tokarev]

Herbert Xu wrote:
> dean gaudet <dean@arctic.org> wrote:
>> in the test program below the getsockname result on a TCP socket changes 
>> across a write which produces EPIPE... here's a fragment of the strace:
>>
>> getsockname(3, {sa_family=AF_INET, sin_port=htons(37636), sin_addr=inet_addr("127.0.0.1")}, [17863593746633850896]) = 0
>> ...
>> write(3, "hi!\n", 4)                    = 4
>> write(3, "hi!\n", 4)                    = -1 EPIPE (Broken pipe)
>> --- SIGPIPE (Broken pipe) @ 0 (0) ---
>> getsockname(3, {sa_family=AF_INET, sin_port=htons(59882), sin_addr=inet_addr("127.0.0.1")}, [16927060683038654480]) = 0
>>
>> why does the port# change?  this is on 2.6.19.1.
> 
> Prior to the last write, the socket entered the CLOSED state meaning
> that the old port is no longer allocated to it.  As a result, the
> last write operates on an unconnected socket which causes a new local
> port to be allocated as an autobind.  It then fails because the socket
> is still not connected.

Well, but why getsockname() didn't just return ENOTCONN?

> So any attempt to run getsockname after an error on the socket is
> simply buggy.

Yes it is.  But so is not returning ENOTCONN from getsockname().  I think.

/mjt

Re: why would EPIPE cause socket port to change?

[Herbert Xu]

On Tue, Jan 23, 2007 at 02:10:39PM +0300, Michael Tokarev wrote:
> 
> Well, but why getsockname() didn't just return ENOTCONN?

It's perfectly valid to have a local port number without being connected.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: why would EPIPE cause socket port to change?

[Michael Tokarev]

Herbert Xu wrote:
> On Tue, Jan 23, 2007 at 02:10:39PM +0300, Michael Tokarev wrote:
>> Well, but why getsockname() didn't just return ENOTCONN?
> 
> It's perfectly valid to have a local port number without being connected.

Er.  You're right - I was confusing getSOCKname() and getPEERname().

Still, after the connection has been closed, there's no chance to do
anything with the filedescriptor but to close it as well, right?  Or
can the fd be reused by making new connection with it, as if it were
just returned from socket() call?

If it's the former, than there's no reason to assign new local address
to it.

/mjt

Re: why would EPIPE cause socket port to change?

[Herbert Xu]

On Tue, Jan 23, 2007 at 02:15:45PM +0300, Michael Tokarev wrote:
> 
> Still, after the connection has been closed, there's no chance to do
> anything with the filedescriptor but to close it as well, right?  Or
> can the fd be reused by making new connection with it, as if it were
> just returned from socket() call?

Yes you can connect out on it again if you're in TCP_CLOSE.  The fact
that the port number has changed implies that you've entered TCP_CLOSE.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: why would EPIPE cause socket port to change?

[Rick Jones]

Herbert Xu wrote:
> dean gaudet <dean@arctic.org> wrote:
> 
>>in the test program below the getsockname result on a TCP socket changes 
>>across a write which produces EPIPE... here's a fragment of the strace:
>>
>>getsockname(3, {sa_family=AF_INET, sin_port=htons(37636), sin_addr=inet_addr("127.0.0.1")}, [17863593746633850896]) = 0
>>...
>>write(3, "hi!\n", 4)                    = 4
>>write(3, "hi!\n", 4)                    = -1 EPIPE (Broken pipe)
>>--- SIGPIPE (Broken pipe) @ 0 (0) ---
>>getsockname(3, {sa_family=AF_INET, sin_port=htons(59882), sin_addr=inet_addr("127.0.0.1")}, [16927060683038654480]) = 0
>>
>>why does the port# change?  this is on 2.6.19.1.
> 
> 
> Prior to the last write, the socket entered the CLOSED state meaning
> that the old port is no longer allocated to it.  As a result, the
> last write operates on an unconnected socket which causes a new local
> port to be allocated as an autobind.  It then fails because the socket
> is still not connected.
> 
> So any attempt to run getsockname after an error on the socket is
> simply buggy.

But falls within the principle of least surprise doesn't it?  Unless the 
application has called close() or bind(), it does seem like a reasonable 
expectation that the port assignments are not changed.

> (fwiw this is one of two reasons i've found for libnss-ldap to leak 
> sockets... causing nscd to crash.)

Of course, that seems rather odd too - why does libnss-ldap check the 
socket name on a socket after an EPIPE anyway?

rick jones

Re: why would EPIPE cause socket port to change?

[dean gaudet]

On Tue, 23 Jan 2007, Rick Jones wrote:

> Herbert Xu wrote:
> > Prior to the last write, the socket entered the CLOSED state meaning
> > that the old port is no longer allocated to it.  As a result, the
> > last write operates on an unconnected socket which causes a new local
> > port to be allocated as an autobind.  It then fails because the socket
> > is still not connected.
> > 
> > So any attempt to run getsockname after an error on the socket is
> > simply buggy.
> 
> But falls within the principle of least surprise doesn't it?  Unless the
> application has called close() or bind(), it does seem like a reasonable
> expectation that the port assignments are not changed.

i sampled a few other OSes...

netbsd returns EINVAL after close
freebsd returns ECONNRESET after close
OSX retains the same port number
solaris 10 returns port 0

actually any of those behaviours seems more appropriate than randomly 
assigning a new port :)  but i like the ENOTCONN suggestion from Michael 
Tokarev the best... it matches the ENOTCONN from getpeername.


> > (fwiw this is one of two reasons i've found for libnss-ldap to leak
> > sockets... causing nscd to crash.)
> 
> Of course, that seems rather odd too - why does libnss-ldap check the socket
> name on a socket after an EPIPE anyway?

libnss-ldap has some code which attempts to determine if its private 
socket has been trampled on in between calls to the library... and to do 
this it caches getsockname/getpeername results and compares them every 
time the library is re-entered... and when there's a mismatch it leaks a 
socket (eventually crashing nscd if you're using that).  i've been trying 
to band-aid over the problem:

http://bugzilla.padl.com/show_bug.cgi?id=304
http://bugzilla.padl.com/show_bug.cgi?id=305

but i'm probably going to need to approach it from another direction -- 
make libnss-ldap monitor the ldap library results so it knows when there's 
been a read/write error so that it stops doing this 
getsockname/getpeername thing after the error has occured.

-dean

Re: why would EPIPE cause socket port to change?

[Stephen Hemminger]

On Tue, 23 Jan 2007 16:44:10 +1100
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> dean gaudet <dean@arctic.org> wrote:
> > in the test program below the getsockname result on a TCP socket changes 
> > across a write which produces EPIPE... here's a fragment of the strace:
> > 
> > getsockname(3, {sa_family=AF_INET, sin_port=htons(37636), sin_addr=inet_addr("127.0.0.1")}, [17863593746633850896]) = 0
> > ...
> > write(3, "hi!\n", 4)                    = 4
> > write(3, "hi!\n", 4)                    = -1 EPIPE (Broken pipe)
> > --- SIGPIPE (Broken pipe) @ 0 (0) ---
> > getsockname(3, {sa_family=AF_INET, sin_port=htons(59882), sin_addr=inet_addr("127.0.0.1")}, [16927060683038654480]) = 0
> > 
> > why does the port# change?  this is on 2.6.19.1.
> 
> Prior to the last write, the socket entered the CLOSED state meaning
> that the old port is no longer allocated to it.  As a result, the
> last write operates on an unconnected socket which causes a new local
> port to be allocated as an autobind.  It then fails because the socket
> is still not connected.

Why does write cause an autobind? One would think that on a
SOCK_STREAM socket, the write should just fail with ENOTCONN


> 
> So any attempt to run getsockname after an error on the socket is
> simply buggy.
> 
> Cheers,


-- 
Stephen Hemminger <shemminger@linux-foundation.org>

Re: why would EPIPE cause socket port to change?

[David Miller]

From: dean gaudet <dean@arctic.org>
Date: Tue, 23 Jan 2007 12:11:01 -0800 (PST)

> libnss-ldap has some code which attempts to determine if its private 
> socket has been trampled on in between calls to the library... and to do 
> this it caches getsockname/getpeername results and compares them every 
> time the library is re-entered... and when there's a mismatch it leaks a 
> socket (eventually crashing nscd if you're using that).  i've been trying 
> to band-aid over the problem:
> 
> http://bugzilla.padl.com/show_bug.cgi?id=304
> http://bugzilla.padl.com/show_bug.cgi?id=305
> 
> but i'm probably going to need to approach it from another direction -- 
> make libnss-ldap monitor the ldap library results so it knows when there's 
> been a read/write error so that it stops doing this 
> getsockname/getpeername thing after the error has occured.

Please do not write programs in this way.  getsockname/getpeername
were never meant to be used in that way, and it's hella inefficient
to keep checking the socket like that to boot.

I really don't see you gaining anything by making this check every
time the user calls into the library.

If the application mucks with the communications channel socket, so
what, it's his application that will go tits up.

Is there some tricky interaction between nscd and something like
libnss-ldap that makes this tom-foolery necessary?

Re: why would EPIPE cause socket port to change?

[Herbert Xu]

On Tue, Jan 23, 2007 at 12:26:52PM -0800, Stephen Hemminger wrote:
> 
> Why does write cause an autobind? One would think that on a
> SOCK_STREAM socket, the write should just fail with ENOTCONN

The autobind is occuring in generic code, i.e., inet_sendmsg().
It will subsequently fail because the socket is not connected.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: why would EPIPE cause socket port to change?

[dean gaudet]

On Tue, 23 Jan 2007, David Miller wrote:

> From: dean gaudet <dean@arctic.org>
> Date: Tue, 23 Jan 2007 12:11:01 -0800 (PST)
> 
> > libnss-ldap has some code which attempts to determine if its private 
> > socket has been trampled on in between calls to the library... and to do 
> > this it caches getsockname/getpeername results and compares them every 
> > time the library is re-entered... and when there's a mismatch it leaks a 
> > socket (eventually crashing nscd if you're using that).  i've been trying 
> > to band-aid over the problem:
> > 
> > http://bugzilla.padl.com/show_bug.cgi?id=304
> > http://bugzilla.padl.com/show_bug.cgi?id=305
> > 
> > but i'm probably going to need to approach it from another direction -- 
> > make libnss-ldap monitor the ldap library results so it knows when there's 
> > been a read/write error so that it stops doing this 
> > getsockname/getpeername thing after the error has occured.
> 
> Please do not write programs in this way.  getsockname/getpeername
> were never meant to be used in that way, and it's hella inefficient
> to keep checking the socket like that to boot.
> 
> I really don't see you gaining anything by making this check every
> time the user calls into the library.
> 
> If the application mucks with the communications channel socket, so
> what, it's his application that will go tits up.
> 
> Is there some tricky interaction between nscd and something like
> libnss-ldap that makes this tom-foolery necessary?
> 

oh heck yeah i totally agree -- it's not my code though, i'm just 
debugging it.

-dean

Re: why would EPIPE cause socket port to change?

[David Miller]

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Wed, 24 Jan 2007 16:58:33 +1100

> On Tue, Jan 23, 2007 at 12:26:52PM -0800, Stephen Hemminger wrote:
> > 
> > Why does write cause an autobind? One would think that on a
> > SOCK_STREAM socket, the write should just fail with ENOTCONN
> 
> The autobind is occuring in generic code, i.e., inet_sendmsg().
> It will subsequently fail because the socket is not connected.

This is one of those situations where the -ENOTCONN might be
preferable, but we are not helping application writers one
iota because the bazillion existing kernels running out there
have the old behavior so app writers have to code for it
anyways.

So I think we should just leave things as they are.

If you read the language Richard Stevens uses when describing
both the usage and implementation of getsockname and
getpeername in BSD, he makes it very clear exactly what these
two interfaces are intended to be used for.

They are to be invoked in situations where the application
has no reason to know what the socket is bound to locally
or remotely, and there are two real cases:

1) determining the result of an auto-bind
2) finding the remote address/port of a socket obtained
   via a fork/exec

I'm sure there are some other minimally different variants,
but those are the two main cases.

This makes it doubly clear that uses such as the one in libnss-ldap
were totally unintended, and should be avoided since every single
OS behaves differently in this case. :-)