SO_REUSEADDR not allowing server and client to use same port

SO_REUSEADDR not allowing server and client to use same port

[Nebojsa Miljanovic]

Hello all,
I have seen similar complaints about this issue before in the list archive, but
I have not seen any resolution. So, I am posting the question again in hope of
getting some replies.

Currently, Linux does not allow reuse of same local port for both TCP server and
TCP client. I understand that there may be a need to prevent two servers from
binding to the same local port, but having 1 server and 1 outgoing client use it
should be allowed. Other Unix like operating systems do allow it. Further more,
Linux SCTP socket API allows for this to happen. And, I can't imagine why this
restriction would exists for TCP and not for SCTP.

This issue is caused by tcp_v4_get_port() function (same check is in IPV6
version of this function). It does does not allow for "tb->fastreuse" bit to be
set when passed socket is in listening state. Suggested code change below would
allow for port reuse by a single listening socket and multiple client sockets.

[neb@mvista ipv4]$ cvs diff -u20 -p -w -b tcp_ipv4.c
Index: tcp_ipv4.c
===================================================================
RCS file:
/cvs/cvsroot/Repository/TelicaRoot/components/mvlinux/cge/devkit/lsp/7xx/linux/net/ipv4/tcp_ipv4.c,v
retrieving revision 1.1.1.2
diff -u -2 -0 -p -w -b -r1.1.1.2 tcp_ipv4.c
--- tcp_ipv4.c  27 Apr 2007 12:34:39 -0000      1.1.1.2
+++ tcp_ipv4.c  28 Feb 2008 18:10:16 -0000
@@ -258,46 +258,54 @@ static int tcp_v4_get_port(struct sock *
        tb = NULL;
        goto tb_not_found;
 tb_found:
        if (!hlist_empty(&tb->owners)) {
                if (sk->sk_reuse > 1)
                        goto success;
                if (tb->fastreuse > 0 &&
                    sk->sk_reuse && sk->sk_state != TCP_LISTEN) {
                        goto success;
                } else {
                        ret = 1;
                        if (tcp_bind_conflict(sk, tb))
                                goto fail_unlock;
                }
        }
 tb_not_found:
        ret = 1;
        if (!tb && (tb = tcp_bucket_create(head, snum)) == NULL)
                goto fail_unlock;
        if (hlist_empty(&tb->owners)) {
+#if 1 /* Do not check for TCP_LISTEN state */
+               if (sk->sk_reuse)
+#else
                if (sk->sk_reuse && sk->sk_state != TCP_LISTEN)
+#endif
                        tb->fastreuse = 1;
                else
                        tb->fastreuse = 0;
        } else if (tb->fastreuse &&
+#if 1 /* Do not check for TCP_LISTEN state */
+                  (!sk->sk_reuse))
+#else
                   (!sk->sk_reuse || sk->sk_state == TCP_LISTEN))
+#endif
                tb->fastreuse = 0;
 success:
        if (!tcp_sk(sk)->bind_hash)
                tcp_bind_hash(sk, tb, snum);
        BUG_TRAP(tcp_sk(sk)->bind_hash == tb);
        ret = 0;

 fail_unlock:
        spin_unlock(&head->lock);
 fail:
        local_bh_enable();
        return ret;
 }


Thanks,
Neb Miljanovic

Alcatel-Lucent

Re: SO_REUSEADDR not allowing server and client to use same port

[Phil Oester]

On Thu, Feb 28, 2008 at 12:15:06PM -0600, Nebojsa Miljanovic wrote:
> Hello all,
> I have seen similar complaints about this issue before in the list archive, but
> I have not seen any resolution. So, I am posting the question again in hope of
> getting some replies.
> 
> Currently, Linux does not allow reuse of same local port for both TCP server and
> TCP client. I understand that there may be a need to prevent two servers from
> binding to the same local port, but having 1 server and 1 outgoing client use it
> should be allowed. Other Unix like operating systems do allow it. Further more,
> Linux SCTP socket API allows for this to happen. And, I can't imagine why this
> restriction would exists for TCP and not for SCTP.

You'd likely get better response by posting this on the netdev list.

Phil

Re: SO_REUSEADDR not allowing server and client to use same port

[Alan Cox]

> Currently, Linux does not allow reuse of same local port for both TCP server and
> TCP client. I understand that there may be a need to prevent two servers from

Think about what happens for simultaneous connect (that obscure little
corner of the tcp/ip spec). Now think about what it means for security.

ALan

Re: SO_REUSEADDR not allowing server and client to use same port

[Willy Tarreau]

On Thu, Feb 28, 2008 at 08:19:26PM +0000, Alan Cox wrote:
> > Currently, Linux does not allow reuse of same local port for both TCP server and
> > TCP client. I understand that there may be a need to prevent two servers from
> 
> Think about what happens for simultaneous connect (that obscure little
> corner of the tcp/ip spec). Now think about what it means for security.
>
> ALan

exactly. That's why I think that requiring the server to explicitly allow
socket reuse is mandatory. On another side of the problem, I really think
that we should make the simultaneous connect configurable. I've been patching
all of my own 2.4 kernels for a while to add a config option to turn it off
because there is a real security issue with this on. At least for mainline,
it should be a sysctl. No stateful firewall I know supports the feature, so
legitimate uses of this feature are likely very very small.

Willy

Re: SO_REUSEADDR not allowing server and client to use same port

[Nebojsa Miljanovic]

Alan,
I would appreciate it if you could elaborate on the security issue. Can you
describe the simultaneous connect scenario?

We have a system we are migrating to Linux from Lynx. We have always had the
ability to re-use ports. We are wondering if this security issue is something we
should take seriously. Or, maybe it is not likely to be seen with our system
running well defined set of applications (i.e. not a PC where anybody could run
anything).

Thanks,
Neb


On 2/28/2008 2:19 PM, Alan Cox wrote:
>>Currently, Linux does not allow reuse of same local port for both TCP server and
>>TCP client. I understand that there may be a need to prevent two servers from
> 
> 
> Think about what happens for simultaneous connect (that obscure little
> corner of the tcp/ip spec). Now think about what it means for security.
> 
> ALan

Re: SO_REUSEADDR not allowing server and client to use same port

[Alan Cox]

On Thu, 13 Mar 2008 14:18:15 -0500
Nebojsa Miljanovic <neb@alcatel-lucent.com> wrote:

> Alan,
> I would appreciate it if you could elaborate on the security issue. Can you
> describe the simultaneous connect scenario?
> 
> We have a system we are migrating to Linux from Lynx. We have always had the
> ability to re-use ports. We are wondering if this security issue is something we
> should take seriously. Or, maybe it is not likely to be seen with our system
> running well defined set of applications (i.e. not a PC where anybody could run
> anything).

See RFC 793. It is permissible for a connection to be created between two
end points both making a connection (crossing SYN frames are both
acked). Thus if you allow such reuse it is possible (although usually
very very hard) to mount a timing based attack.

It's not a very practical attack in most scenarios so we block it out of
correctness and concerns for completely robust and proper behaviour
rather than because it is seen in the wild.

Alan

Re: SO_REUSEADDR not allowing server and client to use same port

[Nebojsa Miljanovic]

Alan,
thanks. With that additional INFO, I was able to find detailed description of
this denial of service attack (attached below).
Just to clarify. Having this port re-use check prevents folks from launching
this attack as opposed to being victim of it?


2.4 Simultaneous Connections
Occasionally, it is possible that hosts XX and YY both wish to establish a
connection and both of them simultaneously initiate the handshake.  This is
called simultaneous connection establishment [RFC 793]. Both hosts XX and YY
send out SYN's to each other. When the SYN's are received, each receiver sends
out a SYN+ACK. Both hosts XX and YY must detect that the SYN and SYN+ACK
actually refer to the same connection. If both hosts XX and YY detect that the
SYN+ACK belongs to the SYN that was recently sent, they switch off the
connection establishment timer and move directly to the SYN_RECVD state. This
flaw could be used to stall a port on a host, using protocols such as FTP where
the server initiates a connection to the client.

As an example, consider (malicious) host XX which has started an (active) FTP
connection to a server YY. XX and YY are connected using the control-port (21 on
YY). YY initiates the connection establishment procedure to initiate data
transfer with XX.

YY sends a SYN to XX, and makes a transition to SYN_SENT state. YY also starts
the connection establishment timer.
XX receives the SYN, and responds with another SYN.
When YY receives the SYN, it assumes that this is a case of a simultaneous open
connection. So, it sends out SYN_ACK to XX, switches off the connection
establishment timer, and transitions to the state SYN_RCVD.
XX receives the SYN_ACK from YY, but does not send a reply.
Since YY is expecting a SYN_ACK in the SYN_RCVD state, and there is no timer, YY
gets stalled in SYN_RCVD state.
XX was able to create a denial-of-service attack.


Neb


On 3/15/2008 8:34 AM, Alan Cox wrote:
> On Thu, 13 Mar 2008 14:18:15 -0500
> Nebojsa Miljanovic <neb@alcatel-lucent.com> wrote:
> 
> 
>>Alan,
>>I would appreciate it if you could elaborate on the security issue. Can you
>>describe the simultaneous connect scenario?
>>
>>We have a system we are migrating to Linux from Lynx. We have always had the
>>ability to re-use ports. We are wondering if this security issue is something we
>>should take seriously. Or, maybe it is not likely to be seen with our system
>>running well defined set of applications (i.e. not a PC where anybody could run
>>anything).
> 
> 
> See RFC 793. It is permissible for a connection to be created between two
> end points both making a connection (crossing SYN frames are both
> acked). Thus if you allow such reuse it is possible (although usually
> very very hard) to mount a timing based attack.
> 
> It's not a very practical attack in most scenarios so we block it out of
> correctness and concerns for completely robust and proper behaviour
> rather than because it is seen in the wild.
> 
> Alan

Re: SO_REUSEADDR not allowing server and client to use same port

[Alan Cox]

On Mon, 17 Mar 2008 11:43:28 -0500
Nebojsa Miljanovic <neb@alcatel-lucent.com> wrote:

> Alan,
> thanks. With that additional INFO, I was able to find detailed description of
> this denial of service attack (attached below).
> Just to clarify. Having this port re-use check prevents folks from launching
> this attack as opposed to being victim of it?

Different issue. I can hijack a connection.

Imagine I have a server bound to *.5000, and someone is about to connect.
If on the server box I am able to bind and issue a connect outwards
matching the inbound connection I will get the connection not the server.

Re: SO_REUSEADDR not allowing server and client to use same port

[Nebojsa Miljanovic]

OK. I see. So, it would have to be some malicious application running together
with the server (i.e. on the same CPU). I do see now why you said it would be
very very hard to make this happen.

Still, it would be nice to introduce SO_REUSEPORT socket options so secure
servers (who happen to be clients as well) can re-use ports when necessary.

Another option would be to check if port re-use is happening inside same
application and allow it. That may make half of the folks happy, so I am not
sure if I like it as much as I like SO_REUSEPORT option.

Thanks,
Neb


On 3/17/2008 12:30 PM, Alan Cox wrote:
> On Mon, 17 Mar 2008 11:43:28 -0500
> Nebojsa Miljanovic <neb@alcatel-lucent.com> wrote:
> 
> 
>>Alan,
>>thanks. With that additional INFO, I was able to find detailed description of
>>this denial of service attack (attached below).
>>Just to clarify. Having this port re-use check prevents folks from launching
>>this attack as opposed to being victim of it?
> 
> 
> Different issue. I can hijack a connection.
> 
> Imagine I have a server bound to *.5000, and someone is about to connect.
> If on the server box I am able to bind and issue a connect outwards
> matching the inbound connection I will get the connection not the server.

Re: SO_REUSEADDR not allowing server and client to use same port

[Willy Tarreau]

On Mon, Mar 17, 2008 at 01:17:29PM -0500, Nebojsa Miljanovic wrote:
> OK. I see. So, it would have to be some malicious application running together
> with the server (i.e. on the same CPU). I do see now why you said it would be
> very very hard to make this happen.
> 
> Still, it would be nice to introduce SO_REUSEPORT socket options so secure
> servers (who happen to be clients as well) can re-use ports when necessary.
> 
> Another option would be to check if port re-use is happening inside same
> application and allow it. That may make half of the folks happy, so I am not
> sure if I like it as much as I like SO_REUSEPORT option.

It is most often useful for hot-restart without interrupting service.
As an example, when I update a config in haproxy, I restart a new instance
which does everything and binds to the ports, and then signals the old one
it may go. That way, there is a complete continuity of service. This is
possible by default on *BSD, and with a (very) little patch on linux.

And I agree with Alan about the security implications. If a service binds
itself with SO_REUSEPORT, it definitely says "I agree to share my connection".
So this is not to be used by default in any random application. But for
servers, it's really useful.

Willy

Re: SO_REUSEADDR not allowing server and client to use same port

[Willy Tarreau]

[cc'd netdev]

On Thu, Feb 28, 2008 at 12:15:06PM -0600, Nebojsa Miljanovic wrote:
> Hello all,
> I have seen similar complaints about this issue before in the list archive, but
> I have not seen any resolution. So, I am posting the question again in hope of
> getting some replies.
> 
> Currently, Linux does not allow reuse of same local port for both TCP server and
> TCP client. I understand that there may be a need to prevent two servers from
> binding to the same local port, but having 1 server and 1 outgoing client use it
> should be allowed. Other Unix like operating systems do allow it.

I've been encountering the need for SO_REUSEPORT as implemented on some BSD
for instance. It allows any process to explicitly permit another one to bind
to the same IP:port provided that it also sets its socket with SO_REUSEPORT.
This is a real requirement when you need to restart a service without any
service disruption.

In the end, I've written a trivial patch for 2.4 (but most likely 2.6 would
use nearly the same one). I noticed that the conflict detection logic already
distinguished between 3 cases: sk->reuse==0, sk->reuse==1, sk->reuse >1.
sk->reuse was set to 1 with SO_REUSEADDR. I simply had to make SO_REUSEPORT
do sk->reuse |= 2 to get everything working as expected.

I don't know yet if it is as easy to do in 2.6, but if there is a possible
acceptance of including this useful feature into mainline, I can try to
work a patch (possibly cleaner than my existing one for 2.4).

Regards,
Willy

Re: SO_REUSEADDR not allowing server and client to use same port

[Nebojsa Miljanovic]

Willy,
I agree. That would also work nicely. I do wish SO_REUSEPORT was available.

But, I am amazed that this issue has a long history (I've seen posts from 1999)
and that there was no progress with it.

Thanks,
Neb


On 2/28/2008 2:36 PM, Willy Tarreau wrote:
> [cc'd netdev]
> 
> On Thu, Feb 28, 2008 at 12:15:06PM -0600, Nebojsa Miljanovic wrote:
> 
>>Hello all,
>>I have seen similar complaints about this issue before in the list archive, but
>>I have not seen any resolution. So, I am posting the question again in hope of
>>getting some replies.
>>
>>Currently, Linux does not allow reuse of same local port for both TCP server and
>>TCP client. I understand that there may be a need to prevent two servers from
>>binding to the same local port, but having 1 server and 1 outgoing client use it
>>should be allowed. Other Unix like operating systems do allow it.
> 
> 
> I've been encountering the need for SO_REUSEPORT as implemented on some BSD
> for instance. It allows any process to explicitly permit another one to bind
> to the same IP:port provided that it also sets its socket with SO_REUSEPORT.
> This is a real requirement when you need to restart a service without any
> service disruption.
> 
> In the end, I've written a trivial patch for 2.4 (but most likely 2.6 would
> use nearly the same one). I noticed that the conflict detection logic already
> distinguished between 3 cases: sk->reuse==0, sk->reuse==1, sk->reuse >1.
> sk->reuse was set to 1 with SO_REUSEADDR. I simply had to make SO_REUSEPORT
> do sk->reuse |= 2 to get everything working as expected.
> 
> I don't know yet if it is as easy to do in 2.6, but if there is a possible
> acceptance of including this useful feature into mainline, I can try to
> work a patch (possibly cleaner than my existing one for 2.4).
> 
> Regards,
> Willy
>