Possible bug when bridging traffic we just SNATed and sent to another router

Possible bug when bridging traffic we just SNATed and sent to another router

[Daniel Collins]

Hi

We have encountered problems when attempting to bridge packets that we
just sent, then saw a second time as the receiving router forwarded
them to another router attached to a different bridge port.

This only occurs when SNAT was used on the original connection, this
seems to prevent the original conntrack entry from being used for the
bridged packet, instead creating a new one for them, our router's
router then sends the replies directly to us (as you'd expect), but
Linux only undoes the first layer of NAT and then discards the packet
since a local socket for it doesn't exist.

This behaviour has been observed under 3.13 and 3.16, is it intended?

We have considered either patching the kernel to undo multiple levels
of NAT, or trying to resolve conntracks using a reversed tuple the
raw/PREROUTING chain using a new iptables target, so that the original
conntrack is used as it is when the SNAT target wasn't used.

Would either of the proposed kernel changes be a
horribly bad idea with unintended side effects? Is there a better way?

Thanks

-- 
Daniel Collins
Software Developer

smoothwall
daniel.collins@smoothwall.com
www.smoothwall.com

Head Office : 1 John Charles Way, Leeds, LS12 6QA, United Kingdom
Tech Office : Eagle Point, Little Park Farm Road, Fareham, PO15 5TD,
United Kingdom
US Office : 8008 Corporate Center Dr #410, Charlotte, NC 28226, United States

Telephone: UK: +44 870-199-9500 US: +1 800-959-3760

Smoothwall Limited is registered in England, Company Number: 4298247
and whose registered address is 1 John Charles Way, Leeds, LS12 6QA
United Kingdom.

Re: Possible bug when bridging traffic we just SNATed and sent to another router

[Florian Westphal]

Daniel Collins <daniel.collins@smoothwall.net> wrote:
> Hi
> 
> We have encountered problems when attempting to bridge packets that we
> just sent, then saw a second time as the receiving router forwarded
> them to another router attached to a different bridge port.

I have no idea what that means.
-v please.

> This only occurs when SNAT was used on the original connection, this
> seems to prevent the original conntrack entry from being used for the
> bridged packet,

Uhh.. what?
Not following, sorry :-/

> instead creating a new one for them, our router's

If a new conntrack is created, then the skb did not (yet) have a conntrack
entry or something has caused the conntrack to be destroyed/discarded.

The latter typically happens with veth, or other
virtualization/container use cases where skb_scrub_packet() is called.

> router then sends the replies directly to us (as you'd expect), but
> Linux only undoes the first layer of NAT and then discards the packet
> since a local socket for it doesn't exist.

again, no idea what that means.  Please explain in more detail or
provide some graph that describes what is connected where, how the
routing happens and where bridge(s) are sitting.

Re: Possible bug when bridging traffic we just SNATed and sent to another router

[Daniel Collins]

+----+   +----+
| R1 |   | R2 |
+----+   +----+
   |       |
+-------------+
| eth0   eth1 |
|   \     /   |
|    \   /    |
|     br0     |
+-------------+

So, the Linux box at the bottom attempts to route a packet from itself
using R1, but R1 forwards the packet on to R2, at which point Linux
creates a new conntrack and rewrites the port number as it passes over
the bridge since that particular IP/port combination was already in
use, by the first conntrack.

Does that make it clearer?

On 5 June 2015 at 15:35, Florian Westphal <fw@strlen.de> wrote:
> Daniel Collins <daniel.collins@smoothwall.net> wrote:
>> Hi
>>
>> We have encountered problems when attempting to bridge packets that we
>> just sent, then saw a second time as the receiving router forwarded
>> them to another router attached to a different bridge port.
>
> I have no idea what that means.
> -v please.
>
>> This only occurs when SNAT was used on the original connection, this
>> seems to prevent the original conntrack entry from being used for the
>> bridged packet,
>
> Uhh.. what?
> Not following, sorry :-/
>
>> instead creating a new one for them, our router's
>
> If a new conntrack is created, then the skb did not (yet) have a conntrack
> entry or something has caused the conntrack to be destroyed/discarded.
>
> The latter typically happens with veth, or other
> virtualization/container use cases where skb_scrub_packet() is called.
>
>> router then sends the replies directly to us (as you'd expect), but
>> Linux only undoes the first layer of NAT and then discards the packet
>> since a local socket for it doesn't exist.
>
> again, no idea what that means.  Please explain in more detail or
> provide some graph that describes what is connected where, how the
> routing happens and where bridge(s) are sitting.

-- 
Daniel Collins
Software Developer

smoothwall
daniel.collins@smoothwall.com
www.smoothwall.com

Head Office : 1 John Charles Way, Leeds, LS12 6QA, United Kingdom
Tech Office : Eagle Point, Little Park Farm Road, Fareham, PO15 5TD,
United Kingdom
US Office : 8008 Corporate Center Dr #410, Charlotte, NC 28226, United States

Telephone: UK: +44 870-199-9500 US: +1 800-959-3760

Smoothwall Limited is registered in England, Company Number: 4298247
and whose registered address is 1 John Charles Way, Leeds, LS12 6QA
United Kingdom.

Re: Possible bug when bridging traffic we just SNATed and sent to another router

[Florian Westphal]

Daniel Collins <daniel.collins@smoothwall.net> wrote:

please avoid top-posting, thanks.

> +----+   +----+
> | R1 |   | R2 |
> +----+   +----+
>    |       |
> +-------------+
> | eth0   eth1 |
> |   \     /   |
> |    \   /    |
> |     br0     |
> +-------------+
> 
> So, the Linux box at the bottom attempts to route a packet from itself
> using R1, but R1 forwards the packet on to R2, at which point Linux
> creates a new conntrack and rewrites the port number as it passes over
> the bridge since that particular IP/port combination was already in
> use, by the first conntrack.
> 
> Does that make it clearer?

So just to make sure:

#1 Linux send skb from (local_addr,daddr)
#2 r1 forwards packet to r2
#3 Linux receives its own packet again, with (saddr local_addr, daddr)
#4 Linux creates a new conntrack entry since lookup for old one would
expect (local_addr is daddr) reply
#5 new conntrack is created, with PAT applied to resolve port clash
collision
#6 remote_addr sends reply, to local_addr
#7 we lookup conntrack, find the one without PAT translation applied
#8 we possibly toss the packet since PAT undo doesn't (yet) yield
skb with a socket.

I'd recommend to fix this setup... If that can't be done, can you
suppress creation of 2nd conntrack entry?

It should be possible via "-t raw -s local_address -j CT --notrack".

Re: Possible bug when bridging traffic we just SNATed and sent to another router

[Daniel Collins]

On 5 June 2015 at 16:08, Florian Westphal <fw@strlen.de> wrote:

> So just to make sure:
>
> #1 Linux send skb from (local_addr,daddr)
> #2 r1 forwards packet to r2
> #3 Linux receives its own packet again, with (saddr local_addr, daddr)
> #4 Linux creates a new conntrack entry since lookup for old one would
> expect (local_addr is daddr) reply
> #5 new conntrack is created, with PAT applied to resolve port clash
> collision
> #6 remote_addr sends reply, to local_addr
> #7 we lookup conntrack, find the one without PAT translation applied
> #8 we possibly toss the packet since PAT undo doesn't (yet) yield
> skb with a socket.

Yes

> I'd recommend to fix this setup... If that can't be done, can you
> suppress creation of 2nd conntrack entry?

This configuration has been used by a small handful of our customers,
we are probably going to get them to fix their routing tables.

> It should be possible via "-t raw -s local_address -j CT --notrack".

Considered this, but doesn't work for us in all cases (tproxy!).

Thanks

-- 
Daniel Collins
Software Developer

smoothwall
daniel.collins@smoothwall.com
www.smoothwall.com

Head Office : 1 John Charles Way, Leeds, LS12 6QA, United Kingdom
Tech Office : Eagle Point, Little Park Farm Road, Fareham, PO15 5TD,
United Kingdom
US Office : 8008 Corporate Center Dr #410, Charlotte, NC 28226, United States

Telephone: UK: +44 870-199-9500 US: +1 800-959-3760

Smoothwall Limited is registered in England, Company Number: 4298247
and whose registered address is 1 John Charles Way, Leeds, LS12 6QA
United Kingdom.

Re: Possible bug when bridging traffic we just SNATed and sent to another router

[Florian Westphal]

Daniel Collins <daniel.collins@smoothwall.net> wrote:
> > So just to make sure:
> >
> > #1 Linux send skb from (local_addr,daddr)
> > #2 r1 forwards packet to r2
> > #3 Linux receives its own packet again, with (saddr local_addr, daddr)
> > #4 Linux creates a new conntrack entry since lookup for old one would
> > expect (local_addr is daddr) reply
> > #5 new conntrack is created, with PAT applied to resolve port clash
> > collision
> > #6 remote_addr sends reply, to local_addr
> > #7 we lookup conntrack, find the one without PAT translation applied
> > #8 we possibly toss the packet since PAT undo doesn't (yet) yield
> > skb with a socket.
> 
> Yes
> 
> > I'd recommend to fix this setup... If that can't be done, can you
> > suppress creation of 2nd conntrack entry?
> 
> This configuration has been used by a small handful of our customers,
> we are probably going to get them to fix their routing tables.
> 
> > It should be possible via "-t raw -s local_address -j CT --notrack".
> 
> Considered this, but doesn't work for us in all cases (tproxy!).

TPROXY doesn't depend on conntrack.

But other than the notrack suggestion i don't have any ideas.