Bug with IPv6-UDP address binding

Bug with IPv6-UDP address binding

[Jesper Dangaard Brouer]

Hi NetDev

I think I have found a problem/bug with IPv6-UDP address binding.

I found this problem while playing with IPVS and IPv6-UDP, but its also
present in more basic/normal situations.

If you have two IPv6 addresses, within the same IPv6 subnet, then one
of the IPv6 addrs takes precedence over the other (for UDP only).

Meaning that, if connecting to the "secondary" IPv6 via UDP, will
result in userspace see/bind the connection as being created to the
"primary" IP, even-though tcpdump shows that the IPv6-UDP packets are
dest the "secondary".

The result is; that only the first IPv6-UDP packet is delivered to
userspace, and the next packets are denied by the kernel as the UDP
socket is "established" with the "primary" IPv6 addr.

I would appreciate some hints to where in the IPv6 code I should look
for this bug.  If any one else wants to fix it, I'm also fine with
that ;-)


Its quite easy to reproduce, using netcat (nc).

Add two addresses to the "server" e.g.:
 ip addr add fee0:cafe::102/64 dev eth0
 ip addr add fee0:cafe::bad/64 dev eth0

Run a netcat listener on "server":
 nc -6 -u -l 2000
(Notice restart the listener between runs, due to limitation in nc)

On the client add an IPv6 addr e.g.:
 ip addr add fee0:cafe::101/64 dev eth0

Run a netcat UDP-IPv6 producer on "client":
  nc -6 -u fee0:cafe::bad 2000

Notice that first packet, will get through, but second packets will
not (nc: Write error: Connection refused).  Running a tcpdump shows
that the kernel is sending back ICMP6, destination unreachable,
unreachable port.

Its also possible to see the problem, simply running "netstat -uan" on
"server", which will show that the "established" UDP connection, is
bound to the wrong "Local Address".

(Tested on both latest net-next kernel at commit 79cda75a1, and also
on RHEL6 approx 2.6.32)

Re: Bug with IPv6-UDP address binding

[Eric Dumazet]

On Wed, 2012-08-08 at 22:37 +0200, Jesper Dangaard Brouer wrote:
> Hi NetDev
> 
> I think I have found a problem/bug with IPv6-UDP address binding.
> 
> I found this problem while playing with IPVS and IPv6-UDP, but its also
> present in more basic/normal situations.
> 
> If you have two IPv6 addresses, within the same IPv6 subnet, then one
> of the IPv6 addrs takes precedence over the other (for UDP only).
> 
> Meaning that, if connecting to the "secondary" IPv6 via UDP, will
> result in userspace see/bind the connection as being created to the
> "primary" IP, even-though tcpdump shows that the IPv6-UDP packets are
> dest the "secondary".
> 
> The result is; that only the first IPv6-UDP packet is delivered to
> userspace, and the next packets are denied by the kernel as the UDP
> socket is "established" with the "primary" IPv6 addr.
> 
> I would appreciate some hints to where in the IPv6 code I should look
> for this bug.  If any one else wants to fix it, I'm also fine with
> that ;-)
> 
> 
> Its quite easy to reproduce, using netcat (nc).
> 
> Add two addresses to the "server" e.g.:
>  ip addr add fee0:cafe::102/64 dev eth0
>  ip addr add fee0:cafe::bad/64 dev eth0
> 
> Run a netcat listener on "server":
>  nc -6 -u -l 2000
> (Notice restart the listener between runs, due to limitation in nc)
> 
> On the client add an IPv6 addr e.g.:
>  ip addr add fee0:cafe::101/64 dev eth0
> 
> Run a netcat UDP-IPv6 producer on "client":
>   nc -6 -u fee0:cafe::bad 2000
> 
> Notice that first packet, will get through, but second packets will
> not (nc: Write error: Connection refused).  Running a tcpdump shows
> that the kernel is sending back ICMP6, destination unreachable,
> unreachable port.
> 
> Its also possible to see the problem, simply running "netstat -uan" on
> "server", which will show that the "established" UDP connection, is
> bound to the wrong "Local Address".
> 
> (Tested on both latest net-next kernel at commit 79cda75a1, and also
> on RHEL6 approx 2.6.32)
> 

Hi Jesper

Thats because the "nc -6 -u -l 2000" on server does :

bind(3, {sa_family=AF_INET6, sin6_port=htons(2000), inet_pton(AF_INET6,
"::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0

recvfrom(3, "\n", 1024, MSG_PEEK, {sa_family=AF_INET6,
sin6_port=htons(53696), inet_pton(AF_INET6, "fee0:cafe::101",
&sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 1

connect(3, {sa_family=AF_INET6, sin6_port=htons(53696),
inet_pton(AF_INET6, "fee0:cafe::101", &sin6_addr), sin6_flowinfo=0,
sin6_scope_id=0}, 28) = 0

And the kernel automatically chooses a SOURCE address (fee0:cafe::102)
that is not what you expected (fee0:cafe::bad)

So its a bug in the application.

UDP connect() is tricky : In this case, nc should learn on what IP
address the client sent the frame. (using recvmsg() and appropriate
ancillary message)

Then nc should bind a new socket on this address, then do the connect()

Re: Bug with IPv6-UDP address binding

[Jesper Dangaard Brouer]

On Wed, 2012-08-08 at 22:59 +0200, Eric Dumazet wrote:
> On Wed, 2012-08-08 at 22:37 +0200, Jesper Dangaard Brouer wrote:
> > Hi NetDev
> > 
> > I think I have found a problem/bug with IPv6-UDP address binding.
> > 
> > I found this problem while playing with IPVS and IPv6-UDP, but its also
> > present in more basic/normal situations.
> > 
> > If you have two IPv6 addresses, within the same IPv6 subnet, then one
> > of the IPv6 addrs takes precedence over the other (for UDP only).
> > 
> > Meaning that, if connecting to the "secondary" IPv6 via UDP, will
> > result in userspace see/bind the connection as being created to the
> > "primary" IP, even-though tcpdump shows that the IPv6-UDP packets are
> > dest the "secondary".
> > 
> > The result is; that only the first IPv6-UDP packet is delivered to
> > userspace, and the next packets are denied by the kernel as the UDP
> > socket is "established" with the "primary" IPv6 addr.
> > 
> > I would appreciate some hints to where in the IPv6 code I should look
> > for this bug.  If any one else wants to fix it, I'm also fine with
> > that ;-)
> > 
> > 
> > Its quite easy to reproduce, using netcat (nc).
> > 
> > Add two addresses to the "server" e.g.:
> >  ip addr add fee0:cafe::102/64 dev eth0
> >  ip addr add fee0:cafe::bad/64 dev eth0
> > 
> > Run a netcat listener on "server":
> >  nc -6 -u -l 2000
> > (Notice restart the listener between runs, due to limitation in nc)
> > 
> > On the client add an IPv6 addr e.g.:
> >  ip addr add fee0:cafe::101/64 dev eth0
> > 
> > Run a netcat UDP-IPv6 producer on "client":
> >   nc -6 -u fee0:cafe::bad 2000
> > 
> > Notice that first packet, will get through, but second packets will
> > not (nc: Write error: Connection refused).  Running a tcpdump shows
> > that the kernel is sending back ICMP6, destination unreachable,
> > unreachable port.
> > 
> > Its also possible to see the problem, simply running "netstat -uan" on
> > "server", which will show that the "established" UDP connection, is
> > bound to the wrong "Local Address".
> > 
> > (Tested on both latest net-next kernel at commit 79cda75a1, and also
> > on RHEL6 approx 2.6.32)
> > 
> 
> Hi Jesper
> 
> Thats because the "nc -6 -u -l 2000" on server does :
> 
> bind(3, {sa_family=AF_INET6, sin6_port=htons(2000), inet_pton(AF_INET6,
> "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
> 
> recvfrom(3, "\n", 1024, MSG_PEEK, {sa_family=AF_INET6,
> sin6_port=htons(53696), inet_pton(AF_INET6, "fee0:cafe::101",
> &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 1
> 
> connect(3, {sa_family=AF_INET6, sin6_port=htons(53696),
> inet_pton(AF_INET6, "fee0:cafe::101", &sin6_addr), sin6_flowinfo=0,
> sin6_scope_id=0}, 28) = 0
> 
> And the kernel automatically chooses a SOURCE address (fee0:cafe::102)
> that is not what you expected (fee0:cafe::bad)

Okay I see.  And this is also the case for IPv4.

Guess I should have read Stephens[1] first, as this problem with
multihomed hosts is described  (on page 219).  He also states, that this
is a problem/feature related to Berkely-derived implementations.  E.g.
Solaris handle this, the way I expected. That is, the source IP address
for the server's reply is the dest IP of the client's request.


> So its a bug in the application.

Yes, I guess its an application bug, because Berkely-derived
implementations don't handle multihomeing well for UDP.

Why are we keeping this, counter-intuitive behavior? 

What about changing the implementation to act like Solaris, which IMHO
makes much more sense?

(BTW, iperf also have this "bug")


> UDP connect() is tricky : In this case, nc should learn on what IP
> address the client sent the frame. (using recvmsg() and appropriate
> ancillary message)

Reading through howto use recvmsg() and parsing of the ancillary
messages.  See [1] "Advanced UDP sockets" page 531-538.  Its quite an
extensive task to extract destination IP address.  No wonder, netcat
missed this part.

> Then nc should bind a new socket on this address, then do the connect()

Yes, after the difficult extraction of the dest IP of the UDP packet.


Now I better understand, why the DNS server named/bind is so annoying,
that is requires a restart after adding IPs.  I guess they didn't
implement this recvmsg(), and instead chooses to bind to all avail IPs
on init/start.

Hints for readers:
For IPv4 is easy to see which is the "secondary" IP via the command "ip
addr" (look for the word "secondary")
For IPv6 I cannot tell which one is the secondary/primary from the "ip
addr" output.  But you can instead do a route lookup via the command
e.g: "ip route get fee0:cafe::102" and look for the "src" field.


[1] UNIX network programming Vol.1 (Networking APIs) by W. Richard
Stevens
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Bug with IPv6-UDP address binding

[Eric Dumazet]

On Thu, 2012-08-09 at 11:40 +0200, Jesper Dangaard Brouer wrote:

> Yes, I guess its an application bug, because Berkely-derived
> implementations don't handle multihomeing well for UDP.
> 
> Why are we keeping this, counter-intuitive behavior? 
> 

Because the BSD api has no other choice.

> What about changing the implementation to act like Solaris, which IMHO
> makes much more sense?
> 
> (BTW, iperf also have this "bug")
> 

Please describe how connect() is able to correctly chose for you a
source address, if you have say 256 addresses.

connect() API only requires the destination.

If you want a precise source, you must use bind() before the connect()

Any UDP application wanting to correctly reflect "correct source IP"
must say so to the kernel.

I dont think kernel could magically help here, really.

Re: Bug with IPv6-UDP address binding

[Eric Dumazet]

On Thu, 2012-08-09 at 11:40 +0200, Jesper Dangaard Brouer wrote:
> On Wed, 2012-08-08 at 22:59 +0200, Eric Dumazet wrote:
> > On Wed, 2012-08-08 at 22:37 +0200, Jesper Dangaard Brouer wrote:

> > Then nc should bind a new socket on this address, then do the connect()
> 
> Yes, after the difficult extraction of the dest IP of the UDP packet.
> 

Thats 10 lines of code. The hard part is in kernel actually ;)

> 
> Now I better understand, why the DNS server named/bind is so annoying,
> that is requires a restart after adding IPs.  I guess they didn't
> implement this recvmsg(), and instead chooses to bind to all avail IPs
> on init/start.

Thats an implementation choice, no more no less.

Here is an IPv4 sample UDP application, able to echo packets with the IP
source set to original DST address of the ping packet.

Doing the same on IPv6 is probably trivial as well

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <linux/udp.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <arpa/inet.h>

#define PORT 4040

int pktinfo_get(struct msghdr *my_hdr, struct in_pktinfo *pktinfo)
{
	int res = -1;

	if (my_hdr->msg_controllen > 0) {
		struct cmsghdr *get_cmsg;
		for (get_cmsg = CMSG_FIRSTHDR(my_hdr); get_cmsg;
			get_cmsg = CMSG_NXTHDR(my_hdr, get_cmsg)) {
			if (get_cmsg->cmsg_type == IP_PKTINFO) {
				struct in_pktinfo *get_pktinfo = (struct in_pktinfo *)CMSG_DATA(get_cmsg);
				memcpy(pktinfo, get_pktinfo, sizeof(*pktinfo));
				res = 0;
			}
		}
	}
	return res;
}

int main(int argc, char *argv[])
{
	int fd = socket(AF_INET, SOCK_DGRAM, 0);
	struct sockaddr_in addr, rem_addr;
	int res, on = 1;
	struct msghdr msghdr;
	struct iovec vec[1];
	char cbuf[512];
	char frame[4096];
	struct in_pktinfo pktinfo;
	int c, count = 1000000;

	while ((c = getopt(argc, argv, "c:")) != -1) {
		if (c == 'c') count = atoi(optarg);
		}
	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_port = htons(PORT);
	if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
		perror("bind");
		return 1;
	}
	setsockopt(fd, SOL_IP, IP_PKTINFO, &on, sizeof(on));

	while (1) {
		memset(&msghdr, 0, sizeof(msghdr));
		msghdr.msg_control = cbuf;
		msghdr.msg_controllen = sizeof(cbuf);
		msghdr.msg_iov = vec;
		msghdr.msg_iovlen = 1;
		vec[0].iov_base = frame;
		vec[0].iov_len = sizeof(frame);
		msghdr.msg_name = &rem_addr;
		msghdr.msg_namelen = sizeof(rem_addr);
		res = recvmsg(fd, &msghdr, 0);
		if (res == -1)
			break;
		if (pktinfo_get(&msghdr, &pktinfo) == 0)
			printf("Got IP_PKTINFO dst addr=%s\n", inet_ntoa(pktinfo.ipi_spec_dst));

		/* ok, just echo reply this frame.
		 * Using sendmsg() will provide IP_PKTINFO back to kernel
		 * to let it use the 'right' source address
		 * (destination address of the incoming packet)
		 */
		vec[0].iov_len = res;
		sendmsg(fd, &msghdr, 0);
		if (--count == 0)
			break;
	}
	return 0;
}

Re: Bug with IPv6-UDP address binding

[Jesper Dangaard Brouer]

On Thu, 2012-08-09 at 13:43 +0200, Eric Dumazet wrote:
> On Thu, 2012-08-09 at 11:40 +0200, Jesper Dangaard Brouer wrote:
> > On Wed, 2012-08-08 at 22:59 +0200, Eric Dumazet wrote:
> > > On Wed, 2012-08-08 at 22:37 +0200, Jesper Dangaard Brouer wrote:
> 
> > > Then nc should bind a new socket on this address, then do the connect()
> > 
> > Yes, after the difficult extraction of the dest IP of the UDP packet.
> > 
> 
> Thats 10 lines of code. 

Really like your example below.  Didn't know about sendmsg(), it sort of
solves the problem, as it provides a method of specifying the source IP.


> The hard part is in kernel actually ;)

Yes, I know.  I was just hoping that we, might have (dst) cached the
info, when the incoming connection was created.  Something like the
sk->sk_dst_cache (without checking... its probably don't make sense)

Its does require a lookup, and I see now, that only having the dest IP
and port, can make this lookup ambiguous.  (and of cause
performance/locking issues... etc.)



> > Now I better understand, why the DNS server named/bind is so annoying,
> > that is requires a restart after adding IPs.  I guess they didn't
> > implement this recvmsg(), and instead chooses to bind to all avail IPs
> > on init/start.
> 
> Thats an implementation choice, no more no less.
> 
> Here is an IPv4 sample UDP application, able to echo packets with the IP
> source set to original DST address of the ping packet.
> 
> Doing the same on IPv6 is probably trivial as well
> 
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <netinet/in.h>
> #include <linux/udp.h>
> #include <string.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <arpa/inet.h>
> 
> #define PORT 4040
> 
> int pktinfo_get(struct msghdr *my_hdr, struct in_pktinfo *pktinfo)
> {
> 	int res = -1;
> 
> 	if (my_hdr->msg_controllen > 0) {
> 		struct cmsghdr *get_cmsg;
> 		for (get_cmsg = CMSG_FIRSTHDR(my_hdr); get_cmsg;
> 			get_cmsg = CMSG_NXTHDR(my_hdr, get_cmsg)) {
> 			if (get_cmsg->cmsg_type == IP_PKTINFO) {
> 				struct in_pktinfo *get_pktinfo = (struct in_pktinfo *)CMSG_DATA(get_cmsg);
> 				memcpy(pktinfo, get_pktinfo, sizeof(*pktinfo));
> 				res = 0;
> 			}
> 		}
> 	}
> 	return res;
> }
> 
> int main(int argc, char *argv[])
> {
> 	int fd = socket(AF_INET, SOCK_DGRAM, 0);
> 	struct sockaddr_in addr, rem_addr;
> 	int res, on = 1;
> 	struct msghdr msghdr;
> 	struct iovec vec[1];
> 	char cbuf[512];
> 	char frame[4096];
> 	struct in_pktinfo pktinfo;
> 	int c, count = 1000000;
> 
> 	while ((c = getopt(argc, argv, "c:")) != -1) {
> 		if (c == 'c') count = atoi(optarg);
> 		}
> 	memset(&addr, 0, sizeof(addr));
> 	addr.sin_family = AF_INET;
> 	addr.sin_port = htons(PORT);
> 	if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
> 		perror("bind");
> 		return 1;
> 	}
> 	setsockopt(fd, SOL_IP, IP_PKTINFO, &on, sizeof(on));
> 
> 	while (1) {
> 		memset(&msghdr, 0, sizeof(msghdr));
> 		msghdr.msg_control = cbuf;
> 		msghdr.msg_controllen = sizeof(cbuf);
> 		msghdr.msg_iov = vec;
> 		msghdr.msg_iovlen = 1;
> 		vec[0].iov_base = frame;
> 		vec[0].iov_len = sizeof(frame);
> 		msghdr.msg_name = &rem_addr;
> 		msghdr.msg_namelen = sizeof(rem_addr);
> 		res = recvmsg(fd, &msghdr, 0);
> 		if (res == -1)
> 			break;
> 		if (pktinfo_get(&msghdr, &pktinfo) == 0)
> 			printf("Got IP_PKTINFO dst addr=%s\n", inet_ntoa(pktinfo.ipi_spec_dst));
> 
> 		/* ok, just echo reply this frame.
> 		 * Using sendmsg() will provide IP_PKTINFO back to kernel
> 		 * to let it use the 'right' source address
> 		 * (destination address of the incoming packet)
> 		 */
> 		vec[0].iov_len = res;
> 		sendmsg(fd, &msghdr, 0);
> 		if (--count == 0)
> 			break;
> 	}
> 	return 0;
> }
> 

Thanks for your input, I value it highly :-)

Re: Bug with IPv6-UDP address binding

[Jesper Dangaard Brouer]

On Thu, 2012-08-09 at 13:43 +0200, Eric Dumazet wrote:

> Here is an IPv4 sample UDP application, able to echo packets with the IP
> source set to original DST address of the ping packet.
> 
> Doing the same on IPv6 is probably trivial as well

I have implemented a combined IPv6 + IPv4 version of the udp echo code,
and placed it on github here:

https://github.com/netoptimizer/network-testing/blob/master/src/udp_echo.c

Comments and patches are welcome ;-)


> #include <sys/types.h>
> #include <sys/socket.h>
> #include <netinet/in.h>
> #include <linux/udp.h>
> #include <string.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <arpa/inet.h>
> 
> #define PORT 4040
> 
> int pktinfo_get(struct msghdr *my_hdr, struct in_pktinfo *pktinfo)
> {
> 	int res = -1;
> 
> 	if (my_hdr->msg_controllen > 0) {
> 		struct cmsghdr *get_cmsg;
> 		for (get_cmsg = CMSG_FIRSTHDR(my_hdr); get_cmsg;
> 			get_cmsg = CMSG_NXTHDR(my_hdr, get_cmsg)) {
> 			if (get_cmsg->cmsg_type == IP_PKTINFO) {
> 				struct in_pktinfo *get_pktinfo = (struct in_pktinfo *)CMSG_DATA(get_cmsg);
> 				memcpy(pktinfo, get_pktinfo, sizeof(*pktinfo));
> 				res = 0;
> 			}
> 		}
> 	}
> 	return res;
> }
> 
> int main(int argc, char *argv[])
> {
> 	int fd = socket(AF_INET, SOCK_DGRAM, 0);
> 	struct sockaddr_in addr, rem_addr;
> 	int res, on = 1;
> 	struct msghdr msghdr;
> 	struct iovec vec[1];
> 	char cbuf[512];
> 	char frame[4096];
> 	struct in_pktinfo pktinfo;
> 	int c, count = 1000000;
> 
> 	while ((c = getopt(argc, argv, "c:")) != -1) {
> 		if (c == 'c') count = atoi(optarg);
> 		}
> 	memset(&addr, 0, sizeof(addr));
> 	addr.sin_family = AF_INET;
> 	addr.sin_port = htons(PORT);
> 	if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
> 		perror("bind");
> 		return 1;
> 	}
> 	setsockopt(fd, SOL_IP, IP_PKTINFO, &on, sizeof(on));
> 
> 	while (1) {
> 		memset(&msghdr, 0, sizeof(msghdr));
> 		msghdr.msg_control = cbuf;
> 		msghdr.msg_controllen = sizeof(cbuf);
> 		msghdr.msg_iov = vec;
> 		msghdr.msg_iovlen = 1;
> 		vec[0].iov_base = frame;
> 		vec[0].iov_len = sizeof(frame);
> 		msghdr.msg_name = &rem_addr;
> 		msghdr.msg_namelen = sizeof(rem_addr);
> 		res = recvmsg(fd, &msghdr, 0);
> 		if (res == -1)
> 			break;
> 		if (pktinfo_get(&msghdr, &pktinfo) == 0)
> 			printf("Got IP_PKTINFO dst addr=%s\n", inet_ntoa(pktinfo.ipi_spec_dst));
> 
> 		/* ok, just echo reply this frame.
> 		 * Using sendmsg() will provide IP_PKTINFO back to kernel
> 		 * to let it use the 'right' source address
> 		 * (destination address of the incoming packet)
> 		 */
> 		vec[0].iov_len = res;
> 		sendmsg(fd, &msghdr, 0);
> 		if (--count == 0)
> 			break;
> 	}
> 	return 0;
> }
> 
> 

Re: Bug with IPv6-UDP address binding

[Eric Dumazet]

On Thu, 2012-08-09 at 11:40 +0200, Jesper Dangaard Brouer wrote:

> Guess I should have read Stephens[1] first, as this problem with
> multihomed hosts is described  (on page 219).  He also states, that this
> is a problem/feature related to Berkely-derived implementations.  E.g.
> Solaris handle this, the way I expected. That is, the source IP address
> for the server's reply is the dest IP of the client's request.

Are you sure ?

So Solaris remembers in socket structure the destination IP address of
last UDP message delivered to userland ?

This is not multi thread safe, so it would be a very bad idea.