any way to let host act as TCP server OR client on same IP/port?

any way to let host act as TCP server OR client on same IP/port?

[Chris Friesen]

I've been asked an interesting question about TCP.  We have some people 
that want to set up a TCP socket that can listen for connections on a 
given IP/port, but also initiate connections from that same IP/port. 
(Only one at a time, of course.)

The TCP state machine seems to allow this (moving from LISTEN to 
SYN_SENT) but it's not a normal transition.

Is there any way to do this using the socket API?

I thought up a hack whereby we could use NFQUEUE to detect an incoming 
SYN and delay it while we call listen() on the socket.  Is there any 
better way to do this?

Thanks,

Chris


-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: any way to let host act as TCP server OR client on same IP/port?

[Eric Dumazet]

Le mercredi 13 juillet 2011 à 10:30 -0600, Chris Friesen a écrit :
> I've been asked an interesting question about TCP.  We have some people 
> that want to set up a TCP socket that can listen for connections on a 
> given IP/port, but also initiate connections from that same IP/port. 
> (Only one at a time, of course.)
> 
> The TCP state machine seems to allow this (moving from LISTEN to 
> SYN_SENT) but it's not a normal transition.
> 
> Is there any way to do this using the socket API?
> 
> I thought up a hack whereby we could use NFQUEUE to detect an incoming 
> SYN and delay it while we call listen() on the socket.  Is there any 
> better way to do this?

Could you try SO_REUSEADDR, on both listener and connect attempt ?

Re: any way to let host act as TCP server OR client on same IP/port?

[Rick Jones]

On 07/13/2011 10:52 AM, Eric Dumazet wrote:
> Le mercredi 13 juillet 2011 à 10:30 -0600, Chris Friesen a écrit :
>> I've been asked an interesting question about TCP.  We have some people
>> that want to set up a TCP socket that can listen for connections on a
>> given IP/port, but also initiate connections from that same IP/port.
>> (Only one at a time, of course.)
>>
>> The TCP state machine seems to allow this (moving from LISTEN to
>> SYN_SENT) but it's not a normal transition.
>>
>> Is there any way to do this using the socket API?
>>
>> I thought up a hack whereby we could use NFQUEUE to detect an incoming
>> SYN and delay it while we call listen() on the socket.  Is there any
>> better way to do this?
>
> Could you try SO_REUSEADDR, on both listener and connect attempt ?

I was thinking the same thing, but it appears to not work under:

raj@tardy:~$ uname -a
Linux tardy 2.6.35-30-generic #54-Ubuntu SMP Tue Jun 7 18:41:54 UTC 2011 
x86_64 GNU/Linux

raj@tardy:~$ ./both
bind active: Address already in use

(nor HP-UX 11.31)

unless I botched the test code somehow:

raj@tardy:~$ cat both.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <arpa/inet.h>
#include <stdlib.h>

int
main(int argc, char *argv[]) {

   struct sockaddr_in me,him;
   int on = 1;
   socklen_t  len;
   int  listener, active, client;


   if ((listener = socket(AF_INET,SOCK_STREAM,0)) < 0) {
     perror("socket listener");
     exit(-1);
   }

   if ((active = socket(AF_INET,SOCK_STREAM,0)) < 0) {
     perror("socket active");
     exit(-1);
   }

   if ((client = socket(AF_INET,SOCK_STREAM,0)) < 0) {
     perror("socket client");
     exit(-1);
   }

   /* setup the listen endpoint */
   memset(&me,0,sizeof(me));

   if (!inet_pton(AF_INET,"127.0.0.1",&(me.sin_addr.s_addr))) {
     perror("inet_pton me");
     exit(-1);
   }
   me.sin_port = htons(23456);
   me.sin_family = AF_INET;

   if (setsockopt(listener,
		 SOL_SOCKET,
		 SO_REUSEADDR,
		 (char *)&on ,
		 sizeof(on)) < 0) {
     perror("setsockopt SO_REUSEADDR listener");
     exit(-1);
   }

   if (bind(listener,
	   (struct sockaddr *)&me,
	   sizeof(me)) < 0) {
     perror("bind listener");
     exit(-1);
   }

   if (listen(listener,128) < 0) {
     perror("listen listener");
     exit(-1);
   }

   /* connect something to it */
   if (connect(client,(struct sockaddr *)&me,sizeof(me)) < 0) {
     perror("connect client");
     exit(-1);
   }

   /* now try to make an active connection from "me" */
   memset(&him,0,sizeof(him));

   if (!inet_pton(AF_INET,"127.0.0.1",&(him.sin_addr.s_addr))) {
     perror("inet_pton him");
     exit(-1);
   }

   him.sin_port = htons(9);  /* assume someone is listening on
					"discard"*/
   him.sin_family = AF_INET;

   if (setsockopt(active,
		 SOL_SOCKET,
		 SO_REUSEADDR,
		 (char *)&on ,
		 sizeof(on)) < 0) {
     perror("setsockopt SO_REUSEADDR active");
     exit(-1);
   }

   if (bind(active,
	   (struct sockaddr *)&me,
	   sizeof(me)) < 0) {
     perror("bind active");
     exit(-1);
   }

   if (connect(active,(struct sockaddr *)&him,sizeof(him)) < 0) {
     perror("connect active");
     exit(-1);
   }

   printf("Successfully created a listen endpoint and an active 
connection from the same IP/port pair\n");

}


rick jones

Re: any way to let host act as TCP server OR client on same IP/port?

[Chris Friesen]

On 07/13/2011 12:05 PM, Rick Jones wrote:
> On 07/13/2011 10:52 AM, Eric Dumazet wrote:
>> Le mercredi 13 juillet 2011 à 10:30 -0600, Chris Friesen a écrit :
>>> I've been asked an interesting question about TCP. We have some people
>>> that want to set up a TCP socket that can listen for connections on a
>>> given IP/port, but also initiate connections from that same IP/port.
>>> (Only one at a time, of course.)
>>>
>>> The TCP state machine seems to allow this (moving from LISTEN to
>>> SYN_SENT) but it's not a normal transition.
>>>
>>> Is there any way to do this using the socket API?
>>>
>>> I thought up a hack whereby we could use NFQUEUE to detect an incoming
>>> SYN and delay it while we call listen() on the socket. Is there any
>>> better way to do this?
>>
>> Could you try SO_REUSEADDR, on both listener and connect attempt ?
> 
> I was thinking the same thing, but it appears to not work under:

<snip>

> if (bind(listener,
> (struct sockaddr *)&me,
> sizeof(me)) < 0) {
> perror("bind listener");
> exit(-1);
> }
> 
> if (listen(listener,128) < 0) {
> perror("listen listener");
> exit(-1);
> }
> 
> /* connect something to it */
> if (connect(client,(struct sockaddr *)&me,sizeof(me)) < 0) {
> perror("connect client");
> exit(-1);

In our case we don't need to actually be connected, just be listening
and ready to either accept() a connection or connect() to someone else.

However, even after removing the connect() call I get:
"bind active: Address already in use"


The TCP state machine shows a single connection going from LISTEN to
SYN_SENT via a "send" operation in the application.  Presumably this
would logically map to a sendto/sendmsg but according to the man page
those don't support specifying addresses for connection-oriented
sockets.  I tried it anyways and got no errors but the following trace
shows that it's dying with SIGPIPE:

bind(3, {sa_family=AF_INET, sin_port=htons(23456), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(3, 128)                          = 0
sendto(3, "\1", 1, 0, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EPIPE (Broken pipe)
--- {si_signo=SIGPIPE, si_code=SI_USER, si_pid=20609, si_uid=8382, si_value={int=2722689790, ptr=0x3ca248f2fe}} (Broken pipe) ---
+++ killed by SIGPIPE +++


Chris



-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: any way to let host act as TCP server OR client on same IP/port?

[Rick Jones]

>>
>> I was thinking the same thing, but it appears to not work under:
>
> <snip>
>
>> if (bind(listener,
>> (struct sockaddr *)&me,
>> sizeof(me))<  0) {
>> perror("bind listener");
>> exit(-1);
>> }
>>
>> if (listen(listener,128)<  0) {
>> perror("listen listener");
>> exit(-1);
>> }
>>
>> /* connect something to it */
>> if (connect(client,(struct sockaddr *)&me,sizeof(me))<  0) {
>> perror("connect client");
>> exit(-1);
>
> In our case we don't need to actually be connected, just be listening
> and ready to either accept() a connection or connect() to someone else.

If one calls listen() against a socket, TCP connections can be 
established at any time, before the accept() calls are made.  As for 
what might happen when one calls connect() on a listen socket when there 
are queued connections awaiting accept() I've no idea.  Presumably they 
would follow the same path as if close() were called against the listen 
endpoint before they were accept()ed.  And if the listen endpoint is 
transitioned to CONNECTED then any subsequently arriving connection 
requests will likely elicit RSTs back to their senders.

If your application's sematics are OK with pending connections being 
dumped, and there is no issue with something else binding to IP,port and 
putting it into the LISTEN state, and if I've read between the lines of 
what you've written correctly, you could simply close() the listen 
socket and create a fresh socket with which to do the connect()?

I think it is only if you need both accepting new connections and 
calling connect() all in parallel that you need enhancements to the stack.

> However, even after removing the connect() call I get:
> "bind active: Address already in use"

Some of this behaviour may come from the "standards" (either formal or 
informal), and what they call for for error returns on the different 
calls under what conditions.

Historically, and not necessarily specific to any one stack, 
SO_REUSEADDR allows one to kill a server application, and restart it and 
get its LISTEN endpoint restarted even while there are old connections 
with IP,port as half the four-tuple (esp TIME_WAITs).  But SO_REUSEADDR 
won't allow the bind(IP,port) to succeed if there is another listen() 
endpoint bound to that IP,port.  That would be an ambiguity.  But, if 
the second endpoint to be bound to IP,port doesn't go listen(), there 
isn't an ambiguity.  This may be where the standards or historical 
expectations come-in if they "require" the bind() fail even if it isn't 
clear there will be a subsequent listen() call against that endpoint.

Presumably, as far as the TCP state machine goes, having one endpoint 
bound only to IP1,Port1 and in the LISTEN state and one simply bound to 
IP,port but not in LISTEN (presumably on its way to a connection 
establishment) is ok.  A arriving segment will hit the four-tuple if it 
exists, and only go to the two-tuple if is in LISTEN and the segment is 
a SYN and won't go to the non-listen two-tuple at all.  But at the API 
level the listen() and connect() calls need to be permitted/able to 
return EADDRINUSE (or something akin) when they then try to create a 
two/four-tuple (listen/connect) that is already in use.

FWIW, the listen() manpage on my Maverick system does list:

ERRORS
        EADDRINUSE
               Another socket is already listening on the same port.

suggesting that the "delay the failure until listen() on the same 
IP,port" is possible, but I don't know if that is vestigial manpage 
cruft or "real."  The manpage for connect() shows EADDRINUSE as well - 
again though I not sure if that is vestigial, so it may not accurately 
convey current stack intent.

> The TCP state machine shows a single connection going from LISTEN to
> SYN_SENT via a "send" operation in the application.  Presumably this
> would logically map to a sendto/sendmsg but according to the man page
> those don't support specifying addresses for connection-oriented
> sockets.  I tried it anyways and got no errors but the following trace
> shows that it's dying with SIGPIPE:
>
> bind(3, {sa_family=AF_INET, sin_port=htons(23456), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
> listen(3, 128)                          = 0
> sendto(3, "\1", 1, 0, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EPIPE (Broken pipe)
> --- {si_signo=SIGPIPE, si_code=SI_USER, si_pid=20609, si_uid=8382, si_value={int=2722689790, ptr=0x3ca248f2fe}} (Broken pipe) ---
> +++ killed by SIGPIPE +++

You might try a connect() call - since T/TCP (vs the even older than 
netperf, ttcp benchmark :) never caught-on, sendto() against a TCP 
endpoint is probably a no-go.

If I'm thinking of the same state machine diagram, the "send" is rather, 
well, generic and does not map directly to any API.

(I tried it very quick and dirty, with no joy)

If your application's sematics are OK with pending connections being 
dumped, and there is no issue with something else
rick

Re: any way to let host act as TCP server OR client on same IP/port?

[Chris Friesen]

On 07/13/2011 02:36 PM, Rick Jones wrote:
>>>
>>> I was thinking the same thing, but it appears to not work under:
>>
>> <snip>
>>
>>> if (bind(listener,
>>> (struct sockaddr *)&me,
>>> sizeof(me))< 0) {
>>> perror("bind listener");
>>> exit(-1);
>>> }
>>>
>>> if (listen(listener,128)< 0) {
>>> perror("listen listener");
>>> exit(-1);
>>> }
>>>
>>> /* connect something to it */
>>> if (connect(client,(struct sockaddr *)&me,sizeof(me))< 0) {
>>> perror("connect client");
>>> exit(-1);
>>
>> In our case we don't need to actually be connected, just be listening
>> and ready to either accept() a connection or connect() to someone else.
>
> If one calls listen() against a socket, TCP connections can be
> established at any time, before the accept() calls are made. As for what
> might happen when one calls connect() on a listen socket when there are
> queued connections awaiting accept() I've no idea.

In Linux at least it appears that you aren't allowed to call listen() 
and then connect().  See below, however.

> If your application's sematics are OK with pending connections being
> dumped, and there is no issue with something else binding to IP,port and
> putting it into the LISTEN state, and if I've read between the lines of
> what you've written correctly, you could simply close() the listen
> socket and create a fresh socket with which to do the connect()?

Ah, of course.  It actually looks like a shutdown() might be sufficient 
to reset the state of the socket enough to allow me to then call 
connect() without needing to create a new socket and bind it.

I'll see what the application guys say.

Thanks,
Chris


-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: any way to let host act as TCP server OR client on same IP/port?

[Chris Friesen]

On 07/13/2011 01:16 PM, Chris Friesen wrote:
> On 07/13/2011 12:05 PM, Rick Jones wrote:
>> On 07/13/2011 10:52 AM, Eric Dumazet wrote:
>>> Le mercredi 13 juillet 2011 à 10:30 -0600, Chris Friesen a écrit :
>>>> I've been asked an interesting question about TCP. We have some people
>>>> that want to set up a TCP socket that can listen for connections on a
>>>> given IP/port, but also initiate connections from that same IP/port.
>>>> (Only one at a time, of course.)
>>>>
>>>> The TCP state machine seems to allow this (moving from LISTEN to
>>>> SYN_SENT) but it's not a normal transition.
>>>>
>>>> Is there any way to do this using the socket API?
>>>>
>>>> I thought up a hack whereby we could use NFQUEUE to detect an incoming
>>>> SYN and delay it while we call listen() on the socket. Is there any
>>>> better way to do this?
>>>
>>> Could you try SO_REUSEADDR, on both listener and connect attempt ?
>>
>> I was thinking the same thing, but it appears to not work under:


> In our case we don't need to actually be connected, just be listening
> and ready to either accept() a connection or connect() to someone else.

It turns out that the application people really do want the server side 
to be able to listen() at the same time as calling connect() from the 
same address/port, so Rick's testcase was accurate.

It would be interesting if linux were to allow this behaviour.  It 
doesn't seem to violate any specs, since we could still return an error 
if we try to connect() to an address that is already connected to us.

Chris


-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: any way to let host act as TCP server OR client on same IP/port?

[Rick Jones]

>> In our case we don't need to actually be connected, just be listening
>> and ready to either accept() a connection or connect() to someone else.
>
> It turns out that the application people really do want the server side
> to be able to listen() at the same time as calling connect() from the
> same address/port, so Rick's testcase was accurate.

So, we are left asking "Why do the application people want that?"  Does 
the server connect() to pseudo-random places, or does it only ever 
connect back to clients which have already connected to it?

If the desire is to have the same well-known port number involved in all 
connections to/from the server (perhaps for simplistic firewall rules 
involving serverip.magicport?) then if the connections are back to the 
clients, the clients could simply open listen endpoints bound to the 
magic port number and the firewall rule become "server IP is source or 
destination) AND (magic port is source or destination)" - assuming the 
clients aren't bind()ing to the magic port number before connect()ing to 
the server.

rick jones

Re: any way to let host act as TCP server OR client on same IP/port?

[Chris Friesen]

>>> In our case we don't need to actually be connected, just be listening
>>> and ready to either accept() a connection or connect() to someone else.
>>
>> It turns out that the application people really do want the server side
>> to be able to listen() at the same time as calling connect() from the
>> same address/port, so Rick's testcase was accurate.
>
> So, we are left asking "Why do the application people want that?" Does
> the server connect() to pseudo-random places, or does it only ever
> connect back to clients which have already connected to it?

I'm not an expert in the application domain, but here's what I know. 
You can think of the situation as having a server, a gateway, and many 
clients.  The goal is to map each client to a single unique port on the 
server-facing side of the gateway, such that all traffic for that client 
goes through that port.

The complicating factor is that traffic may be initiated by either the 
client or the server.  With UDP this isn't a problem, but for TCP it's 
tricky.

For IPv4 the app team originally set it up so that the gateway listens 
on a single port on the server side of the gateway, and when the server 
opens a connection to a client's unique port (with an ephemeral port as 
the source port) it gets NAT'd to that single port.  (The unique ports 
on the server side ensure that there are no 4-tuple collisions.)  For 
traffic originating from the client, the gateway binds a socket to the 
client's unique port and uses it to open a connection to the server.

IPv6 doesn't support NAT (at least not in the stock kernel), so they're 
looking for alternatives.

> If the desire is to have the same well-known port number involved in all
> connections to/from the server (perhaps for simplistic firewall rules
> involving serverip.magicport?)

I suspect it's something like that.

> then if the connections are back to the
> clients, the clients could simply open listen endpoints bound to the
> magic port number and the firewall rule become "server IP is source or
> destination) AND (magic port is source or destination)" - assuming the
> clients aren't bind()ing to the magic port number before connect()ing to
> the server.

We don't control the server, only the gateway.  Ephemeral source ports 
are used for connections opened by the server, so a known port is needed 
on the gateway side.

Chris

-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com