SO_REUSEADDR behavior different from BSD

SO_REUSEADDR behavior different from BSD

[Michael T Kerrisk]

Gidday,

In the classical BSD sockets implementation, the 
SO_REUSEADDR socket option serves two purposes:

(a) to permit a socket to be bound to a port that currently 
    has a bound endpoint in the TIME_WAIT state, and

(b) to allow a terminated server to be restarted (and once
    again be bound to it's well-known port), even if there 
    is a child of the previous server still serving a 
    connection.

On Linux SO_REUSEADDR does serve this purpose.  However, 
other second scenario is mot permitted by SO_REUSEADDR.  
To make it clear what I mean, I’ll detail the scenario.  
Suppose we have the following:

                      Server (host Y)
                    
                      Create listening socket -- fd 4
                      bind() to INADDR_ANY:9999
                      listen()
                      accept()

Client (host X)

Create socket
connect() to Y:9999   <accept unblocks, fd 5 returned>

                      server forks --------+
                                            \ <child>
                      server terminates      closes fd 4
                                             <keeps running, with
                                             fd 5 holding the
                                             connection to client>

At this point, netstat on host Y shows the tuple 

    [local=Y-IP-addr:9999, remote=X-IP-addr:ephem-port] ESTABLISHED

                      New Server started
                      Create listening socket -- fd 4
                      set SO_REUSEADDR to 1 for fd 4

                      bind() fd 4 to port INADDR_ANY:9999
                      WHAT HAPPENS?

On FreeBSD, the bind() succeeds, and then using "netstat –an" 
on host Y shows:

    [local=Y-IP-addr:9999, remote=X-IP-addr:ephem-port] ESTABLISHED
and
    [local=*:9999, remote=*.*] CLOSED

But on Linux (2.6), the bind() fails with EADDRINUSE.

Why does Linux behave differently in this scenario?

Cheers,

Michael

-- 
Michael Kerrisk
mtk-lists@gmx.net

NEU: WLAN-Router für 0,- EUR* - auch für DSL-Wechsler!
GMX DSL = supergünstig & kabellos http://www.gmx.net/de/go/dsl

Re: SO_REUSEADDR behavior different from BSD

[bert hubert]

On Wed, Aug 04, 2004 at 04:25:27PM +0200, Michael T Kerrisk wrote:

> But on Linux (2.6), the bind() fails with EADDRINUSE.

This scenario has previously worked for me. Can you show the exact source
which exhibits this problem?

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: SO_REUSEADDR behavior different from BSD

[Michael T Kerrisk]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="us-ascii", Size: 7086 bytes --]

> On Wed, Aug 04, 2004 at 04:25:27PM +0200, Michael T Kerrisk wrote:
> 
> > But on Linux (2.6), the bind() fails with EADDRINUSE.
> 
> This scenario has previously worked for me. Can you show the exact source
> which exhibits this problem?

Okay, sorry for the delay.  I had to hack together a simpler 
program to demonstrate this.  I've attached the program below 
(it is both server and client, depending on command line options).

Do note that in my original description, on the first time through,
the server did NOT use SO_REUSEADDR.

In other words, using my program, we do the following:

SERVER HOST                  CLIENT HOST  

./reuseaddr_test -s -n
(The above does NOT use 
SO_REUSEADDR)
                             ./reuseaddr_test -c <server-host>

(At this point the server parent exists, but the server
client and child are connected)

./reuseaddr_test -s
(The above DOES use 
SO_REUSEADDR)


Now, on Linux, at this point, the second instance of the 
server fails with EADDRINUSE, even though it did use 
SO_REUSEADDR.  On FreeBSD 5.1, the second server instance 
does successfully bind.

On Linux, if the *first* instance also used SO_REUSEADDR,
then the second instance can bind.  FreeBSD does not require
this.  

I realise that the scenario that delivers EADDRINUSE 
above is slightly unusual, because normally a server always
specifies SO_REUSEADDR, but my question is why the scenario
I'm describing does differ from FreeBSD, which I'm assuming
conforms to the canonical BSD behaviour.

Cheers,

Michael

==

/* reuseaddr_test.c

   Michael Kerrisk, Aug 2004

   IPv4 for simplicity...
*/
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <string.h>
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>

#define DEF_PORT "55555"

#define DEF_SLEEP_TIME 60

#define errMsg(msg)     { perror(msg); }

#define errExit(msg)    { perror(msg); exit(EXIT_FAILURE); }

#define fatalErr(msg)   { fprintf(stderr, "%s\n", msg); \
                          exit(EXIT_FAILURE); }

static void
usageError(char *progName, char *msg)
{
    if (msg != NULL)
        printf("%s", msg);
    fprintf(stderr, "Usage: %s [options] -s \n"
                    "   or: %s [options] -c host\n",
                        progName, progName);
    fprintf(stderr, "Options are:\n");
    fprintf(stderr, "\t-c host   Run as client, connecting to 'host'\n");
    fprintf(stderr, "\t-s        Run as server\n");
    fprintf(stderr, "\t-p port   Use given 'port' number (def=%s)\n",
            DEF_PORT);
    fprintf(stderr, "\t-n        *Don't* use SO_REUSEADDR in server\n");
    fprintf(stderr, "\t-w nsecs  Child sleep interval (def=%d)\n",
            DEF_SLEEP_TIME);
    exit(EXIT_FAILURE);
} /* usageError */

/* Return Internet address as string: "(host, port#)" */

static char *
inet4AddressString(const struct sockaddr_in *addr)
{
    static char buf[1024];
    struct hostent *hent;
    char *host;

    hent = gethostbyaddr((char *) &addr->sin_addr,
                sizeof(addr->sin_addr), addr->sin_family);

    /* Return host name, or dotted address if not found */

    host = (hent != NULL) ? hent->h_name : inet_ntoa(addr->sin_addr);

    snprintf(buf, sizeof(buf), "(%s, %u)", host, ntohs(addr->sin_port));
    return buf;
} /* inet4AddressString */

int
main(int argc, char *argv[])
{
    int opt, sfd, cfd;
    int client, server, reuse;
    char *pstr = DEF_PORT;
    char *host;
    struct sockaddr_in svaddr, claddr;
    struct hostent *h;
    struct in_addr **addrpp;
    char buf[1];
    socklen_t s;
    int optval;
    int stime;
#define CMD_SIZE 1024
    char cmd[CMD_SIZE];

    server = 0;
    client = 0;
    reuse = 1;
    stime = DEF_SLEEP_TIME;

    while ((opt = getopt(argc, argv, "c:sp:nw:")) != -1) {
        switch (opt) {
        case 'c':
            client = 1;
            host = optarg;
            break;

        case 's':
            server = 1;
            break;

        case 'p':
            pstr = optarg;
            break;

        case 'w':
            stime = atoi(optarg);
            break;

        case 'n':
            reuse = 0;
            break;

        default:
            usageError(argv[0], NULL);
            break;
        } /* switch */
    } /* while */

    if ((server && client) || (!server && !client))
        usageError(argv[0], NULL);

    sfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sfd == -1) errExit("socket");

    memset(&svaddr, 0, sizeof(struct sockaddr_in));
    svaddr.sin_family = AF_INET;
    svaddr.sin_port = htons(atoi(pstr));

    if (server) {
        svaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    } else { /* client */
        h = gethostbyname(host);
        if (h == NULL)
            fatalErr("host lookup failed (gethostbyname())");
        addrpp = (struct in_addr **) h->h_addr_list;
        svaddr.sin_addr.s_addr = (*addrpp)->s_addr;
    } 

    if (server) {

        if (reuse) {
            optval = 1;
            if (setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, &optval,
                    sizeof(optval)) == -1) errExit("setsockopt");
        } 

        if (bind(sfd, (struct sockaddr *) &svaddr,
                    sizeof(struct sockaddr_in)) == -1) errExit("bind");

        if (listen(sfd, 5) == -1) errExit("listen");

        printf("Server listen() completed; about to accept()\n");

        s = sizeof(struct sockaddr_in);
        cfd = accept(sfd, (struct sockaddr *) &claddr, &s);
        if (cfd == -1) errExit("accept");

        printf("Server received connection from: %s\n",
                inet4AddressString(&claddr));

        snprintf(cmd, CMD_SIZE, "netstat -an | grep %s", pstr);
        printf("cmd: %s\n", cmd);
        //system(cmd);

        switch(fork()) {
        case -1:
            errExit("fork");

        case 0:
            close(sfd);         /* Close listening socket */
            sleep(2);           /* Give parent a moment to exit */
            printf("Server child started\n");
            system(cmd);
            printf("Server child about to sleep %d seconds\n", stime);

            sleep(stime);

            printf("Server child exiting\n");
            exit(EXIT_FAILURE);

        default:
            printf("Server parent exiting\n");
            exit(EXIT_FAILURE);
        } 

    } else { /* client */

        if (connect(sfd, (struct sockaddr *) &svaddr,
                    sizeof(struct sockaddr_in)) == -1)
            errExit("connect");
        printf("Client connected, now about to read\n");
        read(sfd, buf, 1);
        printf("Client exiting\n");
    } 

    exit(EXIT_SUCCESS);
} /* main */

-- 
Michael Kerrisk
mtk-lists@gmx.net

NEU: WLAN-Router für 0,- EUR* - auch für DSL-Wechsler!
GMX DSL = supergünstig & kabellos http://www.gmx.net/de/go/dsl

Re: SO_REUSEADDR behavior different from BSD

[David Stevens]

Michael,
        Linux does not have SO_REUSEPORT, which post-Net/3
BSD systems do. I believe SO_REUSEADDR on Linux behaves
the same as 4.3BSD, but will differ with 4.4BSD and later because
of the addition of SO_REUSEPORT (unless there's a bug :-)).

                                        +-DLS

Re: SO_REUSEADDR behavior different from BSD

[YOSHIFUJI Hideaki / 吉藤英明]

In article <19686.1091723777@www48.gmx.net> (at Thu, 5 Aug 2004 18:36:17 +0200 (MEST)), "Michael T Kerrisk" <mtk-lists@gmx.net> says:

> Now, on Linux, at this point, the second instance of the 
> server fails with EADDRINUSE, even though it did use 
> SO_REUSEADDR.  On FreeBSD 5.1, the second server instance 
> does successfully bind.

This behavior is intended.
First socket is REQUIRED to set SO_REUSEADDR
I hate BSD's behavior because it is asynmetry.
(Both sockets are required to agree on "REUSEADDR.")

-- 
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA

Re: SO_REUSEADDR behavior different from BSD

[Michael T Kerrisk]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="us-ascii", Size: 911 bytes --]

> In article <19686.1091723777@www48.gmx.net> (at Thu, 5 Aug 2004 18:36:17
> +0200 (MEST)), "Michael T Kerrisk" <mtk-lists@gmx.net> says:
> 
> > Now, on Linux, at this point, the second instance of the 
> > server fails with EADDRINUSE, even though it did use 
> > SO_REUSEADDR.  On FreeBSD 5.1, the second server instance 
> > does successfully bind.
> 
> This behavior is intended.
> First socket is REQUIRED to set SO_REUSEADDR
> I hate BSD's behavior because it is asynmetry.
> (Both sockets are required to agree on "REUSEADDR.")

Hideaki, David,

Thank you for the information / confirmation that this is intended
behavior.  I'll try to see that something on this point makes it 
into the man pages.

Cheers,

Michael

-- 
Michael Kerrisk
mtk-lists@gmx.net

NEU: WLAN-Router für 0,- EUR* - auch für DSL-Wechsler!
GMX DSL = supergünstig & kabellos http://www.gmx.net/de/go/dsl

Re: SO_REUSEADDR behavior different from BSD

[Fernando Gont]

At 11:34 05/08/2004 -0700, YOSHIFUJI Hideaki wrote:

> > Now, on Linux, at this point, the second instance of the
> > server fails with EADDRINUSE, even though it did use
> > SO_REUSEADDR.  On FreeBSD 5.1, the second server instance
> > does successfully bind.
>
>This behavior is intended.
>First socket is REQUIRED to set SO_REUSEADDR
>I hate BSD's behavior because it is asynmetry.
>(Both sockets are required to agree on "REUSEADDR.")

Not sure what you mean by "asymetry".


--
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@acm.org

Re: SO_REUSEADDR behavior different from BSD

[Michael T Kerrisk]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="us-ascii", Size: 999 bytes --]

> At 11:34 05/08/2004 -0700, YOSHIFUJI Hideaki wrote:
> 
> > > Now, on Linux, at this point, the second instance of the
> > > server fails with EADDRINUSE, even though it did use
> > > SO_REUSEADDR.  On FreeBSD 5.1, the second server instance
> > > does successfully bind.
> >
> >This behavior is intended.
> >First socket is REQUIRED to set SO_REUSEADDR
> >I hate BSD's behavior because it is asynmetry.
> >(Both sockets are required to agree on "REUSEADDR.")
> 
> Not sure what you mean by "asymetry".

Ferando,

By asymmetry, I understand Hideaki to mean that BSD requires 
SO_REUSEADDR in the second server bind(), but not the first.
the Linux philosophy then appears to be symmetry: both server
instances must agree that the socket port is reusable (i.e., 
both must set SO_REUSEADDR).

Cheers,

Michael

-- 
Michael Kerrisk
mtk-lists@gmx.net

NEU: WLAN-Router für 0,- EUR* - auch für DSL-Wechsler!
GMX DSL = supergünstig & kabellos http://www.gmx.net/de/go/dsl