MASQUERADE/SNAT and multiple interfaces with the same IP

MASQUERADE/SNAT and multiple interfaces with the same IP

[Kristian Evensen]

Hello,

I am currently experimenting with load-balancing traffic between
multiple tunnels. I have two ip-in-ip tunnels between a router and a
gateway, each tunnel given the same IP in order to simplify address
distribution. In order to route traffic through different tunnels, I
use policy based routing. MASQUERADE/SNAT is used to NAT the packets
coming from the network behind the router.

As long as each flow is sent through the same tunnel, everything works
as expected. However, when I move a flow from one tunnel to another
(for example when a link goes down), there is a difference in behavior
between MASQUERADE and SNAT that I haven't been able to figure out.
When MASQUERADE is used, the NAT mapping is destroyed, one packet is
dropped and then a new mapping is created. With SNAT, this does not
happen and the same mapping is used. The reason keeping the same
mapping on the tunneled packets is important, is to avoid confusing
the remote peer.

After spending long time looking at the source code, I can't figure
out why this happens. Once the MASQUERADE/SNAT rule has been inserted,
to me everything looks the same. One theory I had was that since
MASQUERADE rules are "bound" to an interface, moving the flow to
another interface would cause a new rule to be created and the old one
to eventually time out. However, I always see the DESTROY-message from
conntrack before NEW. I tried tracing the origin of the
DESTROY-message and it seems to be generated by death_by_timeout(). I
have a suspicion that the change of links is detected in early_drop(),
but I have not been able to figure out why.

Does anyone have some hints on where to keep looking, or know the cause?

Thanks in advance for any help,
Kristian

Re: MASQUERADE/SNAT and multiple interfaces with the same IP

[Florian Westphal]

Kristian Evensen <kristian.evensen@gmail.com> wrote:
> After spending long time looking at the source code, I can't figure
> out why this happens. Once the MASQUERADE/SNAT rule has been inserted,
> to me everything looks the same.

The MASQUERADE target zaps conntracks that it "owned" when the
associated link goes down, see masq_device_event() in ipt_MASQUERADE.c

Also, the nat core zaps conntracks when they are sent out on an
interface other than the original interface.

See nf_nat_oif_changed() in iptable_nat.c

Re: MASQUERADE/SNAT and multiple interfaces with the same IP

[Kristian Evensen]

Thank you very much for your reply.

On Fri, Aug 16, 2013 at 1:04 PM, Florian Westphal <fw@strlen.de> wrote:>
> The MASQUERADE target zaps conntracks that it "owned" when the
> associated link goes down, see masq_device_event() in ipt_MASQUERADE.c

I added a pr_info() here to see when this function was called. It is
triggered when the link goes down, as expected, but not when I just
change the mark and move the flows to another interface. I still see
the DESTROY/NEW-behavior though.

> Also, the nat core zaps conntracks when they are sent out on an
> interface other than the original interface.
>
> See nf_nat_oif_changed() in iptable_nat.c

Thanks for letting me know, I will start looking into this function.
Does this mean that even when using SNAT, the old mapping is deleted
and a new created? And does this event not notify conntrack? I have so
far used conntrack -E to monitor changes to nat, but that is perhaps
incorrect?

-Kristian

Re: MASQUERADE/SNAT and multiple interfaces with the same IP

[Florian Westphal]

Kristian Evensen <kristian.evensen@gmail.com> wrote:
> I added a pr_info() here to see when this function was called. It is
> triggered when the link goes down, as expected, but not when I just
> change the mark and move the flows to another interface. I still see
> the DESTROY/NEW-behavior though.

Yes, its handled by iptable_nat.c

> > Also, the nat core zaps conntracks when they are sent out on an
> > interface other than the original interface.
> >
> > See nf_nat_oif_changed() in iptable_nat.c
> 
> Thanks for letting me know, I will start looking into this function.
> Does this mean that even when using SNAT, the old mapping is deleted
> and a new created?

No.

> And does this event not notify conntrack?

It generates a destroy event.

Re: MASQUERADE/SNAT and multiple interfaces with the same IP

[Kristian Evensen]

On Fri, Aug 16, 2013 at 1:18 PM, Florian Westphal <fw@strlen.de> wrote:
>> Does this mean that even when using SNAT, the old mapping is deleted
>> and a new created?
>
> No.

Thank you very much for the help, I see now that I was a little bit
too fast and missed a detail in nf_nat_oif_changed(). If anyone ever
stumbles across the question in the future: SNAT does not set
masq_index and, thus, nf_nat_oif_changed() will always return false.

Have a nice weekend!

-Kristian