Strange MASQUERADING behaviour, bug or feature?

Strange MASQUERADING behaviour, bug or feature?

[Martijn Lievaart]

[-- Attachment #1: Type: text/plain, Size: 3739 bytes --]

Hi,

[ Long story, better to much information than to little ]

I encountered some non-intuitive behaviour with MASQUERADE. I wrote a 
dial-on-demand utility, that uses a tun interface. When it sees packets, 
it brings up the real interface. However, packets are masqueraded with 
the ip address of the tun interface, just like the testsetup below. As 
the real interface is assigned an address with dhcp (the reason for 
using masquerade in the first place), I cannot assign the real address 
to the tun interface.

To reproduce this behaviour in a test setup, I wrote a small utility 
that allocates a nullX device, a device that just discards all packets 
send to it. It assigns a link local ipv4 address to that interface.
On the test machine (machine A, kernel 2.6.15) I set up these iptables 
rules:
*nat
:OUTPUT ACCEPT [1463:59018]
:POSTROUTING ACCEPT [1468:59346]
:PREROUTING ACCEPT [1209:217623]
-A POSTROUTING -o null0 -j MASQUERADE
-A POSTROUTING -o null1 -j MASQUERADE
COMMIT
*filter
:FORWARD ACCEPT [647:54256]
:INPUT ACCEPT [328800:93378200]
:OUTPUT ACCEPT [352242:343481377]
COMMIT

I bring up the first null interface and create a route for the test network:
# ./null
Using interface null0 with address 169.254.0.1.
57: null0: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc pfifo_fast 
qlen 500
    link/[65534]
    inet 169.254.0.1/32 scope global null0
waiting for packets...

(in another window)
# ip ro add 1.2.3.0/24 dev null0 metric 10
The metric is so we can make the packets go to null1 later.

On another machine (B), I set up a route to the test network and ping it.
# ip ro add 1.2.3.0/24 via A
# ping 1.2.3.4

On machine A, I see the packets being masqueraded with the IP address of 
null0:
# tcpdump -i null0 icmp
tcpdump: WARNING: arptype 65534 not supported by libpcap - falling back 
to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on null0, link-type LINUX_SLL (Linux cooked), capture size 96 
bytes
23:36:47.688396 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 4
23:36:48.688383 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 5
23:36:49.688413 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 6
23:36:50.688389 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 7

Now I bring up the second null interface and add a route with a lower 
metric. Note the address assigned to the interface!
# ./null
Using interface null1 with address 169.254.1.1.
58: null1: <POINTOPOINT,MULTICAST,NOARP,UP> mtu 1500 qdisc pfifo_fast 
qlen 500
    link/[65534]
    inet 169.254.1.1/32 scope global null1
waiting for packets...

(in another window)
# ip ro add 1.2.3.0/24 dev null1

The window where I run null1 now starts printing dots, null1 is 
receiving packets. Now when I dump these packets:
# tcpdump -i null1 icmp
tcpdump: WARNING: arptype 65534 not supported by libpcap - falling back 
to cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on null1, link-type LINUX_SLL (Linux cooked), capture size 96 
bytes
23:37:35.688593 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 52
23:37:36.688614 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 53
23:37:37.688626 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 54
23:37:38.688598 IP 169.254.0.1 > 1.2.3.4: icmp 64: echo request seq 55

Hey, that is the wrong source address! Now when I kill the null0 
interface, suddenly the source address changes to the correct 
(169.254.1.1) address. Also when I stop the ping and restart it, the 
source address is correct.

Is this behaviour by design? Is it considered a bug or a feature? Is the 
ping-stream from machine B seen as one contrack entry, and thus causing 
this behaviour?

Regards,
M4


[-- Attachment #2: null.c --]
[-- Type: text/x-csrc, Size: 2308 bytes --]

#include <sys/types.h>
#include <sys/select.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/ioctl.h>

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <linux/if.h>
#include <linux/if_tun.h>

int tun_alloc(char *dev);
void usage(const char *msg);

const char* argv0;
const int debug = 1;

int main(int argc, char **argv)
{
  int fd_tun;
  char null_intf[260];
  char scratch[1024];
  int intf_no;

  argv0 = argv[0];

  if (argc > 1)
    usage(0);

  if (geteuid())
    usage("Must be run by root");

  strcpy(null_intf, "null%d");
  if ((fd_tun = tun_alloc(null_intf)) < 0) {
    perror("Could not alloc tun device");
    exit(1);
  }

  intf_no = atoi(null_intf+4);

  printf("Using interface %s with address 169.254.%d.1.\n", null_intf, intf_no);
  /* Bring up the interface with a link local address */
  sprintf(scratch, "/sbin/ip link set %s up", null_intf);
  system(scratch);
  sprintf(scratch, "/sbin/ip addr add 169.254.%d.1/32 dev %s", intf_no, null_intf);
  system(scratch);
  if (debug) {
    sprintf(scratch, "/sbin/ip addr ls dev %s", null_intf);
    system(scratch);
  }

  debug && puts("waiting for packets...");
  while (1) {
    if (read(fd_tun, scratch, sizeof(scratch))<0) {
      perror("Error reading from tun device");
      exit(1);
    }
    if (debug) {
      printf(".");
      fflush(stdout);
    }
  }    

  /* never get here */
  close(fd_tun);
  exit(0);
}


void usage(const char *msg)
{
  if (msg)
    printf("Error: %s\n\n", msg);

  printf("Null interface, discard everything sent here.\n"
	 "\n"
	 "usage: %s <interface> <route> [<route> ...]\n", argv0);
  exit(1);
}



int tun_alloc(char *dev)
{
  struct ifreq ifr;
  int fd, err;

  if( (fd = open("/dev/net/tun", O_RDWR)) < 0 ) {
    /*    return tun_alloc_old(dev); */
    perror("Cannot open tun allocation device");
    exit(1);
  }

  memset(&ifr, 0, sizeof(ifr));

  /* Flags: IFF_TUN   - TUN device (no Ethernet headers)
   *        IFF_TAP   - TAP device
   *
   *        IFF_NO_PI - Do not provide packet information
   */
  ifr.ifr_flags = IFF_TUN | IFF_NO_PI;
  if( *dev )
    strncpy(ifr.ifr_name, dev, IFNAMSIZ);

  if( (err = ioctl(fd, TUNSETIFF, (void *) &ifr)) < 0 ){
    close(fd);
    return err;
  }
  strcpy(dev, ifr.ifr_name);
  return fd;
}

Re: Strange MASQUERADING behaviour, bug or feature?

[Amin Azez]

Martijn Lievaart wrote:
> Hey, that is the wrong source address! Now when I kill the null0 
> interface, suddenly the source address changes to the correct 
> (169.254.1.1) address. Also when I stop the ping and restart it, the 
> source address is correct.

Thats how NAT works.

The address mappings are setup when the conntrack is created.

You observed that this is less obviously-right for generally stateless 
streams like ping, but it is certainly consistent and makes sense, and 
is neccessary for tcp or udp connections, or they would break everytime 
other routes went up or down.

However consider the ping -I option which lets you choose which iterface 
to ping from; why should it suddenly change interface in the same ping 
just because a different route is up?

Sam

Re: Strange MASQUERADING behaviour, bug or feature?

[Martijn Lievaart]

Amin Azez wrote:

> Martijn Lievaart wrote:
>
>> Hey, that is the wrong source address! Now when I kill the null0 
>> interface, suddenly the source address changes to the correct 
>> (169.254.1.1) address. Also when I stop the ping and restart it, the 
>> source address is correct.
>
>
> Thats how NAT works.
>
> The address mappings are setup when the conntrack is created.
>
> You observed that this is less obviously-right for generally stateless 
> streams like ping, but it is certainly consistent and makes sense, and 
> is neccessary for tcp or udp connections, or they would break 
> everytime other routes went up or down.


For SNAT, I can understand this. For MASQUERADE, this is probably also 
the correct behaviour. However, why is this "ping stream" seen as one 
contrack?

But other than that, yes I see now that I cannot use this form  of dial 
on demand with MASQUERADING. Too bad.

Thanks,
M4

Re: Strange MASQUERADING behaviour, bug or feature?

[Yasuyuki KOZAKAI]

From: Martijn Lievaart <m@rtij.nl>
Date: Wed, 19 Apr 2006 18:28:03 +0200

> Amin Azez wrote:
> 
> > Martijn Lievaart wrote:
> >
> >> Hey, that is the wrong source address! Now when I kill the null0 
> >> interface, suddenly the source address changes to the correct 
> >> (169.254.1.1) address. Also when I stop the ping and restart it, the 
> >> source address is correct.
> >
> >
> > Thats how NAT works.
> >
> > The address mappings are setup when the conntrack is created.
> >
> > You observed that this is less obviously-right for generally stateless 
> > streams like ping, but it is certainly consistent and makes sense, and 
> > is neccessary for tcp or udp connections, or they would break 
> > everytime other routes went up or down.
> 
> 
> For SNAT, I can understand this. For MASQUERADE, this is probably also 
> the correct behaviour. However, why is this "ping stream" seen as one 
> contrack?

The echo requests/replies with same ID, same source - destination address
combination belongs same conntrack. The identity of conntrack is
unrelated with input/output interfaces.

Probably ping kept to use same source address for echo requests on your
test, and routing changed output interface according to routing table.

If source address before masquerading was changed, ip_conntrack would
create different conntrack and MASQUERADE target would use it.

That's why tcpdump saw changed source address after you restart ping,
I think.

-- Yasuyuki Kozakai

Re: Strange MASQUERADING behaviour, bug or feature?

[Amin Azez]

Martijn Lievaart wrote:

> Amin Azez wrote:
>
>> Martijn Lievaart wrote:
>>
>>> Hey, that is the wrong source address! Now when I kill the null0 
>>> interface, suddenly the source address changes to the correct 
>>> (169.254.1.1) address. Also when I stop the ping and restart it, the 
>>> source address is correct.
>>
>>
>>
>> Thats how NAT works.
>>
>> The address mappings are setup when the conntrack is created.
>>
>> You observed that this is less obviously-right for generally 
>> stateless streams like ping, but it is certainly consistent and makes 
>> sense, and is neccessary for tcp or udp connections, or they would 
>> break everytime other routes went up or down.
>
>
>
> For SNAT, I can understand this. For MASQUERADE, this is probably also 
> the correct behaviour. 

BTW MASQ is SNAT but where you specify the interface instead of the IP 
address, useful where a different IP address is given for each connection.

> However, why is this "ping stream" seen as one contrack?
>
> But other than that, yes I see now that I cannot use this form  of 
> dial on demand with MASQUERADING. Too bad.

I think that if you could, it would cause you other problems, it is 
harmful/useless to change the source address of TCP connections partway 
through.

Perhaps you could explain what problem you are trying to solve with this 
method.

Sam

Re: Strange MASQUERADING behaviour, bug or feature?

[Martijn Lievaart]

Amin Azez zei:

> I think that if you could, it would cause you other problems, it is
> harmful/useless to change the source address of TCP connections partway
> through.

It is also useless to reroute a tcp connection through a different
interface with different masquerading while keeping the old masquerading,
so in either case, your tcp connection will not work anymore. If however
only SYNs are sent up till now, changing the source address WOULD work
(from a tcp point of view, not from a conntrack pov).

My thinko was that changing the route (output interface) would make MASQ
tear down the connections, just like when the interface went down and then
up.

> Perhaps you could explain what problem you are trying to solve with this
> method.

Dial on demand, when pppd doesn't cut it. And combine that with
masquerading. It seems to work on 2.6 when I do:
- add masquerading rules for both real and dummy interface
- create dummy interface and route with high metric
- while forever
  - if packets arrive at the dummy interface
    - bring up the real interface
    - add route with normal metric
    - destroy and recreate the dummy interface (so the contracks are
destroyed)

This works more or less, which is enough for me at the moment.
Unfortunately, it does not work on my 2.4 setup, but that's probably a
different issue related to tun/tap  (cannot destroy the dummy interface,
it just lives on).

M4

Re: Strange MASQUERADING behaviour, bug or feature?

[Martijn Lievaart]

Amin Azez zei:
> I don't have an answer, unless you can use the --tee option of ipt_route
> to divert a copy of packets to your dummy interface.
> If you use ipt_condition with this, then you can control when it happens
> with a /proc file.

That's a good idea, I'll try that!

M4