UDP sockets bound to ANY send answers with wrong src ip address

UDP sockets bound to ANY send answers with wrong src ip address

[Denis Vlasenko]

I observe that UDP sockets listening on ANY
send response packets with ip addr derived from
ip address of interface which is used to send 'em
instead of using dst ip address of client's packet.

I was bitten by this with DNS and NTP.

This is a test setup:

{net 172.16/16}----- 172.16.22.2 [multihomed] 195.66.192.167 ----....

This is a UDP server on multihomed box:
# echo test | nc -vvvnu -l -p 222
listening on [any] 222 ...
connect to [172.16.22.2] from (UNKNOWN) [172.16.42.177] 333
ping
 sent 5, rcvd 5 : Connection refused
#

This is client on 172.16.42.177:
# echo ping | nc -vvvnu -p 333 195.66.192.167 222
(UNKNOWN) [195.66.192.167] 222 (?) open

it waits forever, never getting 'test' string from server.

This is a tcpdump of this event:
13:58:00.555207 172.16.42.177.333 > 195.66.192.167.222: udp 5 (DF)
13:58:00.559849 172.16.22.2.222 > 172.16.42.177.333: udp 5 (DF)
                ^^^^^^^^^^^
13:58:00.560457 172.16.42.177 > 172.16.22.2: icmp: 172.16.42.177 udp port 333 unreachable [tos 0xc0]

This is why 172.16.42.177 thinks that port 333 is unreachable:
# lsof -nP | grep 333
ntpd        418   root  mem    REG        3,4   133316      68184 /.share/usr/lib/all-lib/libreadline.so.3.0
most       7559   root  txt    REG        3,4    46828      24333 /.share/usr/app/most-4.9.2/bin/most
nc        15085   root    3u  IPv4      29381                 UDP 172.16.42.177:333->195.66.192.167:222
#                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

kernel does not think that 172.16.22.2.222 > 172.16.42.177.333 packet
belong to this socket, and it is right imo.

Test works ok if I remove -u from both netcats (i.e. TCP works).

client:
# uname -a
Linux firebird 2.6.6-rc1-bk1 #1 Wed May 26 11:03:11 EEST 2004 i686 unknown

server:
# uname -a
Linux oldie 2.4.22-rc2_QoS #1 Tue Nov 18 15:55:00 GMT-2 2003 i586 unknown
root@oldie:/.share/home/vda#

I saw this behavior on 2.6 based machines as well.
-- 
vda

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Julian Anastasov]

	Hello,

On Wed, 9 Jun 2004, Denis Vlasenko wrote:

> I observe that UDP sockets listening on ANY
> send response packets with ip addr derived from
> ip address of interface which is used to send 'em
> instead of using dst ip address of client's packet.
>
> I was bitten by this with DNS and NTP.

	The problem is in the apps. You have some options:

- use sockets listening on ANY and using sendmsg with IP_PKTINFO
and providing the desired src IP in ipi_spec_dst

- you can also create many listener sockets listening on
particular src IP and to reply using the socket where the
request is received, because the kernel does not know any
association between request and reply packets if the listener
is bound to ANY.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Denis Vlasenko]

On Wednesday 09 June 2004 15:18, Julian Anastasov wrote:
> Hello,
>
> On Wed, 9 Jun 2004, Denis Vlasenko wrote:
> > I observe that UDP sockets listening on ANY
> > send response packets with ip addr derived from
> > ip address of interface which is used to send 'em
> > instead of using dst ip address of client's packet.
> >
> > I was bitten by this with DNS and NTP.
>
> 	The problem is in the apps. You have some options:
>
> - use sockets listening on ANY and using sendmsg with IP_PKTINFO
> and providing the desired src IP in ipi_spec_dst
>
> - you can also create many listener sockets listening on
> particular src IP and to reply using the socket where the
> request is received, because the kernel does not know any
> association between request and reply packets if the listener
> is bound to ANY.

Aha! That's why ntpd is so eager to bind to everything imaginable!
(Which does not help BTW with non-permanent interfaces like
ppp, tun etc).

/me googling for that IP_PKTINFO thing...
--
vda

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Martijn van Oosterhout]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

On Wed, Jun 09, 2004 at 02:25:39PM +0300, Denis Vlasenko wrote:
> I observe that UDP sockets listening on ANY
> send response packets with ip addr derived from
> ip address of interface which is used to send 'em
> instead of using dst ip address of client's packet.
> 
> I was bitten by this with DNS and NTP.

This is the responsibility of the program. Unless the UDP packet is
bound to a particular address, or the program specifies an IP, the
kernel will pick one. For this reason both the BIND and NTP daemons
open a socket for each interface so they can control this. netstat on
my machine shows:

udp        0      0 192.168.1.225:123       0.0.0.0:*                           
udp        0      0 127.0.0.1:123           0.0.0.0:*                           
udp        0      0 0.0.0.0:123             0.0.0.0:*                        

for the NTP server. The DNS has similar machanism. Remember, UDP
doesn't involve connections, so there is no concept of "replying" to a
packet, the program has to manage that itself.

In your example, if you tell netcat which address to bind to, it will
work.

Hope this helps,
-- 
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: UDP sockets bound to ANY send answers with wrong src ip address

[YOSHIFUJI Hideaki / 吉藤英明]

In article <200406091425.39324.vda@port.imtp.ilyichevsk.odessa.ua> (at Wed, 9 Jun 2004 14:25:39 +0300), Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> says:

> I observe that UDP sockets listening on ANY
> send response packets with ip addr derived from
> ip address of interface which is used to send 'em
> instead of using dst ip address of client's packet.

use IP_PKTINFO when responding the client.

--yoshfuji

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Denis Vlasenko]

On Wednesday 09 June 2004 15:24, YOSHIFUJI Hideaki wrote:
> Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> says:
> > I observe that UDP sockets listening on ANY
> > send response packets with ip addr derived from
> > ip address of interface which is used to send 'em
> > instead of using dst ip address of client's packet.
>
> use IP_PKTINFO when responding the client.

Thanks!
With your help and some googling I've found and adapted 
code to get dst ip of UDP packet.

Small test program successfully ran and reported correct
dst addresses of incoming UDP packets.

Now, I am trying to fix (or shall I say 'improve'?) dnscache.
You may find some code below my sig. It's a start.

The problem is, how to _send replies_ with correct src ip?
I can bind a temporary socket to needed src address,
do a sendto(), then close socket. This will work,
but this can introduce a race - any incoming
packet to this (ip,port) will inadvertently
be classified as belonging to temp socket!
This is going to be a nasty bug, manifesting
itself only under load.

I looked into sendmsg(). Looks like ther is no way to
indicate source ip.

Shall I use some other technique?
--
vda

#if defined IP_RECVDSTADDR
# define DSTADDR_SOCKOPT IP_RECVDSTADDR
# define DSTADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_addr)))
# define dstaddr(x) (CMSG_DATA(x))
#elif defined IP_PKTINFO
# define DSTADDR_SOCKOPT IP_PKTINFO
# define DSTADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_pktinfo)))
# define dstaddr(x) (&(((struct in_pktinfo *)(CMSG_DATA(x)))->ipi_addr))
#else
# error "can't determine socket option"
#endif

int socket_recv4_dst(int s,char *buf,int len,char ip[4],uint16 *port, char ipdst[4])
{
  int r;

  struct iovec iov[1];
  struct sockaddr_in sa;
  union control_data cmsg;
  struct cmsghdr *cmsgptr;
  struct msghdr msg;

  iov[0].iov_base = buf;
  iov[0].iov_len = len;

  memset(&msg, 0, sizeof msg);
  msg.msg_name = &sa;
  msg.msg_namelen = sizeof sa;
  msg.msg_iov = iov;
  msg.msg_iovlen = 1;
  msg.msg_control = &cmsg;
  msg.msg_controllen = sizeof cmsg;

  { // FIXME: we need to do it ONCE! move it into socket_bind4_dstaddropt()
    int sockopt;
    sockopt = 1;
    if (setsockopt(s, IPPROTO_IP, DSTADDR_SOCKOPT, &sockopt, sizeof sockopt) == -1)
      return -1;
  }

  //r = recvfrom(s,buf,len,0,(struct sockaddr *) &sa,&dummy);
  r = recvmsg(s, &msg, 0);
  if (r == -1) return -1;
  // Here we retrieve destination IP and memorize it
  for (cmsgptr = CMSG_FIRSTHDR(&msg);
  cmsgptr != NULL;
  cmsgptr = CMSG_NXTHDR(&msg, cmsgptr)) {
    if (cmsgptr->cmsg_level == IPPROTO_IP
    && cmsgptr->cmsg_type == DSTADDR_SOCKOPT) {
      byte_copy(ipdst,4,(char *) dstaddr(cmsgptr));
    }
  }

  return r;
}

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Herbert Xu]

Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> wrote:
> 
> I looked into sendmsg(). Looks like ther is no way to
> indicate source ip.
> 
> Shall I use some other technique?

IP_PKTINFO works just as well there.  Look at the ip_cmsg_send call
in udp_sendmsg.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email:  Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Denis Vlasenko]

On Friday 11 June 2004 13:34, Herbert Xu wrote:
> Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> wrote:
> > I looked into sendmsg(). Looks like ther is no way to
> > indicate source ip.
> >
> > Shall I use some other technique?
>
> IP_PKTINFO works just as well there.  Look at the ip_cmsg_send call
> in udp_sendmsg.

int udp_sendmsg(...)
{
...
        ipc.addr = inet->saddr;

        ipc.oif = sk->sk_bound_dev_if;
        if (msg->msg_controllen) {
                err = ip_cmsg_send(msg, &ipc);
                if (err)
                        return err;
                if (ipc.opt)
                        free = 1;
                connected = 0;
        }
        if (!ipc.opt)
                ipc.opt = inet->opt;

        saddr = ipc.addr;
...

int ip_cmsg_send(struct msghdr *msg, struct ipcm_cookie *ipc)
{
...
                case IP_PKTINFO:
                {
                        struct in_pktinfo *info;
                        if (cmsg->cmsg_len != CMSG_LEN(sizeof(struct in_pktinfo)))
                                return -EINVAL;
                        info = (struct in_pktinfo *)CMSG_DATA(cmsg);
                        ipc->oif = info->ipi_ifindex;
                        ipc->addr = info->ipi_spec_dst.s_addr;

manpage:
IP_PKTINFO
Pass an IP_PKTINFO ancillary message that contains a pktinfo structure that supplies some information  about  the
incoming packet. This only works for datagram oriented sockets.

struct in_pktinfo
{
    unsigned int   ipi_ifindex;  /* Interface index */
    struct in_addr ipi_spec_dst; /* Routing destination address */
    struct in_addr ipi_addr;     /* Header Destination address */
};

Hmmm... do I have to set a *routing dest address* field
to set src ip address of my UDP packet?
-- 
vda

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Julian Anastasov]

	Hello,

On Fri, 11 Jun 2004, Denis Vlasenko wrote:

> Hmmm... do I have to set a *routing dest address* field
> to set src ip address of my UDP packet?

	Try such function:

static int my_send(int fd, unsigned srcip, (struct sockaddr *) remote,
			char *data, int len)
{
	struct iovec iov = { data, len };
	struct {
		struct cmsghdr cm;
		struct in_pktinfo ipi;
	} cmsg = {
		.cm = {
			.cmsg_len	= sizeof(struct cmsghdr) +
						sizeof(struct in_pktinfo),
			.cmsg_level	= SOL_IP,
			.cmsg_type	= IP_PKTINFO,
		},
		.ipi = {
			.ipi_ifindex	= 0,
			.ipi_spec_dst	= srcip,
		},
	};
	struct msghdr m = {
		.msg_name	= remote,
		.msg_namelen	= sizeof(struct sockaddr_in),
		.msg_iov	= &iov,
		.msg_iovlen	= 1,
		.msg_control	= &cmsg,
		.msg_controllen	= sizeof(cmsg),
		.msg_flags	= 0,
	};

	return sendmsg(fd, &m, MSG_NOSIGNAL|MSG_DONTWAIT);
}

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: UDP sockets bound to ANY send answers with wrong src ip address

[Denis Vlasenko]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

On Friday 11 June 2004 16:53, Julian Anastasov wrote:
> 	Hello,
>
> On Fri, 11 Jun 2004, Denis Vlasenko wrote:
> > Hmmm... do I have to set a *routing dest address* field
> > to set src ip address of my UDP packet?
>
> 	Try such function:
>
> [snip]

Thanks for your excellent help Julian.
It helped me to finally make my code work.

Pity I did not keep it that way, because 
it wouldn't be portable. I converted it
to 'man cmsg' compliant way.

On Friday 11 June 2004 17:14, Henrik Nordstrom wrote:
> [snip]
> > I am trying to figure it out, currently I'm getting
> > -EINVAL in sendmsg in this code:
> >
> > # define SRCADDR_SOCKOPT IP_PKTINFO
> > # define SRCADDR_DATASIZE (CMSG_SPACE(sizeof(struct in_pktinfo)))
> > # define srcaddr(x) (&(((struct in_pktinfo
> > *)(CMSG_DATA(x)))->ipi_spec_dst))
>
> Drop the defines. These are only obfuscating your code and the source to
> your error. You need to pass a in_pktinfo structure in the message, and

They were there in the example recv code from the Net. That code tried
to use IP_DSTADDR (IIRC) option (a BSDism or something) if it is #defined.

Dropped #defines.

I ended up with following patch for djbdns. Run tested.

Thanks again for your help guys.
--
vda

[-- Attachment #2: d.diff --]
[-- Type: text/x-diff, Size: 6291 bytes --]

diff -u djbdns-1.05.orig/dnscache.c djbdns-1.05.new/dnscache.c
--- djbdns-1.05.orig/dnscache.c	Sun Feb 11 23:11:45 2001
+++ djbdns-1.05.new/dnscache.c	Sun Jun 13 00:51:52 2004
@@ -62,6 +62,7 @@
   iopause_fd *io;
   char ip[4];
   uint16 port;
+  char myip[4];
   char id[2];
 } u[MAXUDP];
 int uactive = 0;
@@ -78,7 +79,7 @@
   if (!u[j].active) return;
   response_id(u[j].id);
   if (response_len > 512) response_tc();
-  socket_send4(udp53,response,response_len,u[j].ip,u[j].port);
+  socket_send4_from(udp53,response,response_len,u[j].ip,u[j].port,u[j].myip);
   log_querydone(&u[j].active,response_len);
   u[j].active = 0; --uactive;
 }
@@ -109,7 +110,7 @@
   x = u + j;
   taia_now(&x->start);
 
-  len = socket_recv4(udp53,buf,sizeof buf,x->ip,&x->port);
+  len = socket_recv4_rememberdst(udp53,buf,sizeof buf,x->ip,&x->port,x->myip);
   if (len == -1) return;
   if (len >= sizeof buf) return;
   if (x->port < 1024) if (x->port != 53) return;
diff -u djbdns-1.05.orig/socket.h djbdns-1.05.new/socket.h
--- djbdns-1.05.orig/socket.h	Sun Feb 11 23:11:45 2001
+++ djbdns-1.05.new/socket.h	Sun Jun 13 00:51:44 2004
@@ -8,8 +8,8 @@
 
 extern int socket_connect4(int,const char *,uint16);
 extern int socket_connected(int);
-extern int socket_bind4(int,char *,uint16);
-extern int socket_bind4_reuse(int,char *,uint16);
+extern int socket_bind4(int,const char *,uint16);
+extern int socket_bind4_reuse(int,const char *,uint16);
 extern int socket_listen(int,int);
 extern int socket_accept4(int,char *,uint16 *);
 extern int socket_recv4(int,char *,int,char *,uint16 *);
@@ -18,5 +18,8 @@
 extern int socket_remote4(int,char *,uint16 *);
 
 extern void socket_tryreservein(int,int);
+
+int socket_recv4_rememberdst(int s,char *buf,int len,char ip[4],uint16 *port, char ipdst[4]);
+int socket_send4_from(int s,const char *buf,int len,const char ip[4],uint16 port,const char from[4]);
 
 #endif
diff -u djbdns-1.05.orig/socket_bind.c djbdns-1.05.new/socket_bind.c
--- djbdns-1.05.orig/socket_bind.c	Sun Feb 11 23:11:45 2001
+++ djbdns-1.05.new/socket_bind.c	Fri Jun 11 12:49:45 2004
@@ -5,7 +5,7 @@
 #include "byte.h"
 #include "socket.h"
 
-int socket_bind4(int s,char ip[4],uint16 port)
+int socket_bind4(int s,const char ip[4],uint16 port)
 {
   struct sockaddr_in sa;
 
@@ -17,7 +17,7 @@
   return bind(s,(struct sockaddr *) &sa,sizeof sa);
 }
 
-int socket_bind4_reuse(int s,char ip[4],uint16 port)
+int socket_bind4_reuse(int s,const char ip[4],uint16 port)
 {
   int opt = 1;
   setsockopt(s,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof opt);
diff -u djbdns-1.05.orig/socket_recv.c djbdns-1.05.new/socket_recv.c
--- djbdns-1.05.orig/socket_recv.c	Sun Feb 11 23:11:45 2001
+++ djbdns-1.05.new/socket_recv.c	Sun Jun 13 00:51:48 2004
@@ -2,6 +2,7 @@
 #include <sys/param.h>
 #include <sys/socket.h>
 #include <netinet/in.h>
+#include <string.h>	/* memset() */
 #include "byte.h"
 #include "socket.h"
 
@@ -16,6 +17,57 @@
 
   byte_copy(ip,4,(char *) &sa.sin_addr);
   uint16_unpack_big((char *) &sa.sin_port,port);
+
+  return r;
+}
+
+int socket_recv4_rememberdst(int s,char *buf,int len,char ip[4],uint16 *port, char ipdst[4])
+{
+  int r;
+
+  struct iovec iov[1];
+  struct sockaddr_in remote;
+  char cmsg[CMSG_SPACE(sizeof(struct in_pktinfo))];
+  struct cmsghdr *cmsgptr;
+  struct msghdr msg;
+
+  iov[0].iov_base = buf;
+  iov[0].iov_len = len;
+
+  memset(&msg, 0, sizeof(msg));
+  msg.msg_name = &remote;
+  msg.msg_namelen = sizeof(remote);
+  msg.msg_iov = iov;
+  msg.msg_iovlen = 1;
+  msg.msg_control = &cmsg;
+  msg.msg_controllen = sizeof(cmsg);
+
+  { // FIXME: do it only once
+    int sockopt;
+    sockopt = 1;
+    if (setsockopt(s, SOL_IP, IP_PKTINFO, &sockopt, sizeof sockopt) == -1)
+      return -1;
+  }
+
+  r = recvmsg(s, &msg, 0);
+  if (r == -1) return -1;
+
+  byte_copy(ip, 4, (char*)&remote.sin_addr);
+  uint16_unpack_big((char*)&remote.sin_port, port);
+
+  /* Here we try to retrieve destination IP and memorize it */
+  byte_copy(ipdst, 4, "\0\0\0" ); /* 4th '\0' char is also there */
+  for (cmsgptr = CMSG_FIRSTHDR(&msg);
+  cmsgptr != NULL;
+  cmsgptr = CMSG_NXTHDR(&msg, cmsgptr)) {
+    if (cmsgptr->cmsg_level == SOL_IP
+    && cmsgptr->cmsg_type == IP_PKTINFO) {
+
+#define pktinfo(cmsgptr) ( (struct in_pktinfo*)(CMSG_DATA(cmsgptr)) )
+
+      byte_copy(ipdst, 4, (char*)(&pktinfo(cmsgptr)->ipi_addr) );
+    }
+  }
 
   return r;
 }
diff -u djbdns-1.05.orig/socket_send.c djbdns-1.05.new/socket_send.c
--- djbdns-1.05.orig/socket_send.c	Sun Feb 11 23:11:45 2001
+++ djbdns-1.05.new/socket_send.c	Sun Jun 13 00:46:40 2004
@@ -2,6 +2,7 @@
 #include <sys/param.h>
 #include <sys/socket.h>
 #include <netinet/in.h>
+#include <string.h>	/* memset() */
 #include "byte.h"
 #include "socket.h"
 
@@ -15,4 +16,53 @@
   byte_copy((char *) &sa.sin_addr,4,ip);
 
   return sendto(s,buf,len,0,(struct sockaddr *) &sa,sizeof sa);
+}
+
+int socket_send4_from(int s,const char *buf,int len,const char ip[4],uint16 port,const char from[4])
+{
+  /* man recvmsg and man cmsg is needed to make sense of code below */
+
+  struct sockaddr_in remote;
+  struct iovec iov[1];
+  struct msghdr msg;
+  char cbuf[CMSG_SPACE(sizeof(struct in_pktinfo))];
+  struct cmsghdr* cmsgptr;
+  struct in_pktinfo* pktptr;
+
+  memset(&remote, 0, sizeof(remote));
+  remote.sin_family = AF_INET;
+  uint16_pack_big( (char*)&remote.sin_port, port);
+  byte_copy( (char*)&remote.sin_addr, 4, ip);
+
+  iov[0].iov_base = buf;
+  iov[0].iov_len = len;
+
+  memset(cbuf, 0, sizeof(cbuf));
+
+  memset(&msg, 0, sizeof(msg));
+  msg.msg_name = &remote;
+  msg.msg_namelen = sizeof(remote);
+  msg.msg_iov = iov;
+  msg.msg_iovlen = 1;
+  msg.msg_control = cbuf;
+  msg.msg_controllen = sizeof(cbuf);
+
+  cmsgptr = CMSG_FIRSTHDR(&msg);
+  cmsgptr->cmsg_level = SOL_IP;
+  cmsgptr->cmsg_type = IP_PKTINFO;
+  cmsgptr->cmsg_len = CMSG_LEN(sizeof(struct in_pktinfo));
+
+#define pktinfo(cmsgptr) ((struct in_pktinfo *)(CMSG_DATA(cmsgptr)))
+
+  pktptr = pktinfo(cmsgptr);
+  /* pktptr->ipi_ifindex = 0;   -- already done by memset(cbuf...) */
+  byte_copy( (char*)(&pktptr->ipi_spec_dst), 4, from);
+
+  { // FIXME: do it only once
+    int sockopt = 1;
+    if (setsockopt(s, SOL_IP, IP_PKTINFO, &sockopt, sizeof sockopt) == -1)
+      return -1;
+  }
+
+  return sendmsg(s, &msg, 0);
 }