connect(2) reassociation regression

connect(2) reassociation regression

[William Ahern]

I've stumbled upon what may be a regression in connect(2) behavior.

My DNS library uses connect(2) to reassociate UDP sockets. That way the
kernel can filter my packets, and it makes for cleaner code overall. The
Linux manual page makes it pretty clear that this is okay, and at least one
interpretation of POSIX (certainly the one I had) does as well.

At some point in the 3.x cycle (maybe after 3.2.0) something was changed.
Whereas previously any reassociation worked, regardless of destination
network, now if the _first_ association is to the loopback, any subsequent
association to non-loopback fails with EINVAL. However, if the loopback is
the second or later association then everything continues to work. In other
words, the sequence

	connect(127.0.0.1), connect(8.8.8.8)

fails with EINVAL, but

	connect(8.8.8.8), connect(127.0.0.1), connect(1.2.3.4)

succeeds.

I admit that originally I simply presumed that on each reassociation the
kernel would handle reassociating the source address in addition to the
destination address. The technique worked everywhere I tested, including
Linux, Solaris, NetBSD, OpenBSD, FreeBSD. And I should note that it even
worked when reassociating to different external networks (and still works on
everything but Linux, AFAICT).

I realize now that arguably POSIX only requires that a second connnect call
change the destination address, and not the source address. But what would
be the point of allowing a reassociation if the source address is never
changed? Because any two addresses may route to entirely different networks
or over different devices, the capability to reassociate would be pointless.

OTOH, if you explicitly called bind before connect, most systems these days
will unbind the source address when reassociating. That may be undesirable
behavior, but it is long-standing behavior AFAICT, including on Linux. One
way to bypass the new Linux behavior is to reset the socket with
connect(AF_UNSPEC), but under the pedantic interpretation of POSIX that's
not guaranteed to work.

I first posted this issue on comp.unix.programmer, including example code.
The thread is at

https://groups.google.com/forum/?fromgroups=#!topic/comp.unix.programmer/ya0V-rr8ip0

Although it's hard to follow w/ Google's horrendous interface.

Re: connect(2) reassociation regression

[Eric Dumazet]

On Fri, 2013-03-15 at 22:56 -0700, William Ahern wrote:
> I've stumbled upon what may be a regression in connect(2) behavior.
> 
> My DNS library uses connect(2) to reassociate UDP sockets. That way the
> kernel can filter my packets, and it makes for cleaner code overall. The
> Linux manual page makes it pretty clear that this is okay, and at least one
> interpretation of POSIX (certainly the one I had) does as well.
> 
> At some point in the 3.x cycle (maybe after 3.2.0) something was changed.
> Whereas previously any reassociation worked, regardless of destination
> network, now if the _first_ association is to the loopback, any subsequent
> association to non-loopback fails with EINVAL. However, if the loopback is
> the second or later association then everything continues to work. In other
> words, the sequence
> 
> 	connect(127.0.0.1), connect(8.8.8.8)
> 
> fails with EINVAL, but
> 
> 	connect(8.8.8.8), connect(127.0.0.1), connect(1.2.3.4)
> 
> succeeds.
> 
> I admit that originally I simply presumed that on each reassociation the
> kernel would handle reassociating the source address in addition to the
> destination address. The technique worked everywhere I tested, including
> Linux, Solaris, NetBSD, OpenBSD, FreeBSD. And I should note that it even
> worked when reassociating to different external networks (and still works on
> everything but Linux, AFAICT).
> 
> I realize now that arguably POSIX only requires that a second connnect call
> change the destination address, and not the source address. But what would
> be the point of allowing a reassociation if the source address is never
> changed? Because any two addresses may route to entirely different networks
> or over different devices, the capability to reassociate would be pointless.
> 
> OTOH, if you explicitly called bind before connect, most systems these days
> will unbind the source address when reassociating. That may be undesirable
> behavior, but it is long-standing behavior AFAICT, including on Linux. One
> way to bypass the new Linux behavior is to reset the socket with
> connect(AF_UNSPEC), but under the pedantic interpretation of POSIX that's
> not guaranteed to work.

There is an issue as the connect() call sets both local address:port and
remote address, in the case the local address was not already set by a
prior bind().

And once bound to a local address, its not really clear if we are
allowed to bind to a different one, and fall in the possible traps of
SO_REUSEADDR and find another socket bound to the same local addr:port.

So if the second connect() also change the source port, I am pretty sure
some applications will badly break.

I would just avoid the problem of handling this mess, and let the
application close the socket and allocate a new one.

Changing the kernel behavior on these kind of unspecified stuff might
break some other applications.

Clearly the BSD API was bad, as the connect() is a 'super operation',
not only setting the remote address:port, but also the local
address:port given the current routing table.

Re: connect(2) reassociation regression

[William Ahern]

On Sat, Mar 16, 2013 at 07:51:07AM -0700, Eric Dumazet wrote:
> On Fri, 2013-03-15 at 22:56 -0700, William Ahern wrote:
> > I've stumbled upon what may be a regression in connect(2) behavior.
<snip>
> > At some point in the 3.x cycle (maybe after 3.2.0) something was changed.
> > Whereas previously any reassociation worked, regardless of destination
> > network, now if the _first_ association is to the loopback, any subsequent
> > association to non-loopback fails with EINVAL. However, if the loopback is
> > the second or later association then everything continues to work. In other
> > words, the sequence
> > 
<snip>
> 
> There is an issue as the connect() call sets both local address:port and
> remote address, in the case the local address was not already set by a
> prior bind().
> 
> And once bound to a local address, its not really clear if we are
> allowed to bind to a different one, and fall in the possible traps of
> SO_REUSEADDR and find another socket bound to the same local addr:port.
> 
> So if the second connect() also change the source port, I am pretty sure
> some applications will badly break.
> 
> I would just avoid the problem of handling this mess, and let the
> application close the socket and allocate a new one.

This is for UDP, not TCP. The alternative is sendto/recvfrom. For a
non-blocking, recursive DNS library reopening sockets all of the time is
really messy.

> Changing the kernel behavior on these kind of unspecified stuff might
> break some other applications.

Indeed. The behavior _was_ changed, and my application broke, and
potentially many other applications.

> Clearly the BSD API was bad, as the connect() is a 'super operation',
> not only setting the remote address:port, but also the local
> address:port given the current routing table.

Well, AFAICT Linux implemented this behavior for years. FWIW, Solaris also
implements this behavior. It's been universal until recently.