Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

[Neil Horman]

Hey all-

	Got a question regarding expected/desired behavior of $SUBJECT


I have a report of a problem with a program that opens two sockets:

The first socket is UDP and binds to 127.0.0.1 on a randomly selected port

The second socket is UDP and calls connect, sending to the first socket

Both sockets are part of the same process and have SO_REUSEADDR set

After the connect the second socket sends a message to the first socket.  The
first socket waits for the message by calling select().

Its observed that occasionally the first socket fails to receive the message,
which is odd, given that the system is unloaded, and this is the only message
being sent.  A little investigation shows that when this happens, the client and
the server wind up bound to the same port.

This happens because the second socket calls inet_autobind during the connect
call, and since both it and the server have SO_REUSEADDR set, it is possible
that the autobind will select the same port that the first socket is bound to.
When this happens the sendmsg path can get confused.  Specifically, when the skb
is delivered to the destination socket, the hash lookup might find the wrong
entry and enqueue the skb to the second socket instead of the first.

Questions:

1) Is that expected?

2) If not, what do you think the best way to fix it is?

	a) Deny autobinds to the same port when SO_REUSEADDR is set, but allow
explicity binds to the same port?

	b) Deny both autobinds and explicit binds to the same port/addr,
effectively disablind SO_REUSEADDR with UDP, kind of like with listening TCP
sockets

	c) Add magic to udp_rcv to detect skbs originating from local sockets,
and _dont_ deliver to the socket it originated from

I'm inclined to say, no this is not expected behavior, and that we should fix it
with option A, but I'm interested in getting other opinions before I go down any
particular path.

Thanks!
Neil

Re: Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

[Eric Dumazet]

Le vendredi 19 novembre 2010 à 19:48 -0500, Neil Horman a écrit :
> Hey all-
> 
> 	Got a question regarding expected/desired behavior of $SUBJECT
> 
> 
> I have a report of a problem with a program that opens two sockets:
> 
> The first socket is UDP and binds to 127.0.0.1 on a randomly selected port
> 
> The second socket is UDP and calls connect, sending to the first socket
> 
> Both sockets are part of the same process and have SO_REUSEADDR set
> 
> After the connect the second socket sends a message to the first socket.  The
> first socket waits for the message by calling select().
> 
> Its observed that occasionally the first socket fails to receive the message,
> which is odd, given that the system is unloaded, and this is the only message
> being sent.  A little investigation shows that when this happens, the client and
> the server wind up bound to the same port.
> 
> This happens because the second socket calls inet_autobind during the connect
> call, and since both it and the server have SO_REUSEADDR set, it is possible
> that the autobind will select the same port that the first socket is bound to.
> When this happens the sendmsg path can get confused.  Specifically, when the skb
> is delivered to the destination socket, the hash lookup might find the wrong
> entry and enqueue the skb to the second socket instead of the first.
> 
> Questions:
> 
> 1) Is that expected?
> 

Is SO_REUSADDR used on both sockets ?

May I ask why SO_REUSEADDR is set in the first place on UDP sockets ?

I use it before a bind() on a given port (non null), but apparently your
program doesnt bind() the 2nd socket before its connect() ?


> 2) If not, what do you think the best way to fix it is?
> 
> 	a) Deny autobinds to the same port when SO_REUSEADDR is set, but allow
> explicity binds to the same port?
> 
> 	b) Deny both autobinds and explicit binds to the same port/addr,
> effectively disablind SO_REUSEADDR with UDP, kind of like with listening TCP
> sockets
> 
> 	c) Add magic to udp_rcv to detect skbs originating from local sockets,
> and _dont_ deliver to the socket it originated from

Why ? Its a valid use case IMHO, even with a single socket.

> 
> I'm inclined to say, no this is not expected behavior, and that we should fix it
> with option A, but I'm interested in getting other opinions before I go down any
> particular path.
> 

autobind certainly is a problem, we tried to 'fix' it in recent past and
had to revert some patches. We tried to allow more sockets to be used
but we failed.

Re: Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

[Neil Horman]

On Sat, Nov 20, 2010 at 05:06:55AM +0100, Eric Dumazet wrote:
> Le vendredi 19 novembre 2010 à 19:48 -0500, Neil Horman a écrit :
> > Hey all-
> > 
> > 	Got a question regarding expected/desired behavior of $SUBJECT
> > 
> > 
> > I have a report of a problem with a program that opens two sockets:
> > 
> > The first socket is UDP and binds to 127.0.0.1 on a randomly selected port
> > 
> > The second socket is UDP and calls connect, sending to the first socket
> > 
> > Both sockets are part of the same process and have SO_REUSEADDR set
> > 
> > After the connect the second socket sends a message to the first socket.  The
> > first socket waits for the message by calling select().
> > 
> > Its observed that occasionally the first socket fails to receive the message,
> > which is odd, given that the system is unloaded, and this is the only message
> > being sent.  A little investigation shows that when this happens, the client and
> > the server wind up bound to the same port.
> > 
> > This happens because the second socket calls inet_autobind during the connect
> > call, and since both it and the server have SO_REUSEADDR set, it is possible
> > that the autobind will select the same port that the first socket is bound to.
> > When this happens the sendmsg path can get confused.  Specifically, when the skb
> > is delivered to the destination socket, the hash lookup might find the wrong
> > entry and enqueue the skb to the second socket instead of the first.
> > 
> > Questions:
> > 
> > 1) Is that expected?
> > 
> 
> Is SO_REUSADDR used on both sockets ?
> 
Yes, both udp sockets have SO_REUSEADDR set on them

> May I ask why SO_REUSEADDR is set in the first place on UDP sockets ?
> 
Honestly, I don't know.  This was reported to me as part of:
https://bugzilla.redhat.com/show_bug.cgi?id=643911
At first the consensus was that this bug was fixed by your patch series that
adds a secondary hash for udp sockets, but on closer inspection it appears that
this is just a case of what I described above.  Specifically, that two sockets
are inadvertently binding to the same port/address, and as such when someone
sends from socket A to socket B, A is actually the socket that receives the
frame rather than socket B (as the program might have intended).

> I use it before a bind() on a given port (non null), but apparently your
> program doesnt bind() the 2nd socket before its connect() ?
> 
Correct, the second socket is autobound, via inet_autobind as called from
connect(), which we call on the second udp socket.  When that happens the socket
is bound to a random port.  But sometimes if the socket has SO_REUSEADDR set it
winds up binding to the same port that the first socket is bound to, resulting
in the above problem.

> 
> > 2) If not, what do you think the best way to fix it is?
> > 
> > 	a) Deny autobinds to the same port when SO_REUSEADDR is set, but allow
> > explicity binds to the same port?
> > 
> > 	b) Deny both autobinds and explicit binds to the same port/addr,
> > effectively disablind SO_REUSEADDR with UDP, kind of like with listening TCP
> > sockets
> > 
> > 	c) Add magic to udp_rcv to detect skbs originating from local sockets,
> > and _dont_ deliver to the socket it originated from
> 
> Why ? Its a valid use case IMHO, even with a single socket.
> 
> > 
> > I'm inclined to say, no this is not expected behavior, and that we should fix it
> > with option A, but I'm interested in getting other opinions before I go down any
> > particular path.
> > 
> 
> autobind certainly is a problem, we tried to 'fix' it in recent past and
> had to revert some patches. We tried to allow more sockets to be used
> but we failed.
> 
Agreed.  My thought was to add logic to udp_lib_lport_inuse such that, if
sk_reuse is set on both sockets, and the input snum is 0 (indicating autobind)
we should not allow binding sk to inet_sk(sk2)->num.  Thoughts?
Neil

> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

[Eric Dumazet]

Le samedi 20 novembre 2010 à 10:04 -0500, Neil Horman a écrit :
> https://bugzilla.redhat.com/show_bug.cgi?id=643911

I am not allowed to see this bug report.

' You are not authorized to access bug #643911 '

Re: Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

[Eric Dumazet]

Le samedi 20 novembre 2010 à 10:04 -0500, Neil Horman a écrit :

> Agreed.  My thought was to add logic to udp_lib_lport_inuse such that, if
> sk_reuse is set on both sockets, and the input snum is 0 (indicating autobind)
> we should not allow binding sk to inet_sk(sk2)->num.  Thoughts?

I dont know, problem is this could be possible right now if sk2 is bound
on 127.0.0.2 address. Adding this test would reduce possible space.

Autobind is tricky, it chooses a port while address is part of the
problem.

Some loopback scalability problems could be solved if full range of
127.0.0.0/8 addresses were used, avoiding the dst refcount false
sharing :)

Re: Question regarding expected behavior of two udp sockets with SO_REUSEADDR set

[Neil Horman]

On Sat, Nov 20, 2010 at 04:44:47PM +0100, Eric Dumazet wrote:
> Le samedi 20 novembre 2010 à 10:04 -0500, Neil Horman a écrit :
> 
> > Agreed.  My thought was to add logic to udp_lib_lport_inuse such that, if
> > sk_reuse is set on both sockets, and the input snum is 0 (indicating autobind)
> > we should not allow binding sk to inet_sk(sk2)->num.  Thoughts?
> 
> I dont know, problem is this could be possible right now if sk2 is bound
> on 127.0.0.2 address. Adding this test would reduce possible space.
> 
Well, yes, that is in fact the exact problem that I origionally described.  Both
sockets are bound to the same port on 127.0.0.1 on the same system.  Since both
sockets have SO_REUSEADDR set, this is allowed, but when either socket sends  a
messaage, the delivery code does a lookup in __udp4_lib_lookup, and the sock
structure that gets returned there is ambiguous. It could be either of the two
bound sockets, and the result will depend on which order they exist in the
hslot->head list (since they will both by definition hash to the same bucket).

Another solution might be to treat daddr of 127 specially, and force a traversal
of the entire hslot->head list, delivering to every matching packet, but we
might need to do that for all locally owned daddrs, so I'm not sure thats a
great solution.  Other thoughts welcome, including just leave it all well enough
alone :)


> Autobind is tricky, it chooses a port while address is part of the
> problem.
> 
Agreed, I'm comming to understand that :)

Neil