randomly SNATed devices after reboot

randomly SNATed devices after reboot

[Ethy H. Brito]

Hi All

I have this setup in which there are lots of static IPs "SNATed" IP-Phones
behind a Linux machine. A very simply NAT machine. Just one SNAT rule for the
phones' network.

At every Linux machine reboot, some of those phones, randomly, simply does not
register at some outside-nat SIP server.

Investigating with tcpdump I can see, at the external interface, "not snated"
packets from those not registered phones. Packets from the other phones are
correctly "snatted".

Rebooting the Linux machine scatters this behavior among the phones: some are
randomly registered and some not. Rebooting the phone, and just the phone
itself, does not change anything.

Some background I think relevant:

	1) The Linux ip address is added (one interface, two IPs in two
	different nets) further during boot, at rc.local, immediately before
	the SNAT rule; No NAT rule was added up to this point.

	2) if I change the ip address, under the same netmask, of any
	non-registered phone, it registers immediately; But this does not assure
	it will register again after a new Linux reboot. In fact it may not
	register again after that. Already happened.

	3) All IP-Phones have "keep alive SIP connection" active.

I have a suspicious about what is going on: some race condition. 
But I'd like your thoughts.

Thanx in advance

Regards

Ethy

Re: randomly SNATed devices after reboot

[Vigneswaran R]

On 05/15/2014 07:12 PM, Ethy H. Brito wrote:
> Hi All
>
> I have this setup in which there are lots of static IPs "SNATed" IP-Phones
> behind a Linux machine. A very simply NAT machine. Just one SNAT rule for the
> phones' network.
>
> At every Linux machine reboot, some of those phones, randomly, simply does not
> register at some outside-nat SIP server.
>
> Investigating with tcpdump I can see, at the external interface, "not snated"
> packets from those not registered phones. Packets from the other phones are
> correctly "snatted".

May be, some phones are trying to register via ESTABLISHED connections 
which not getting SNATed. So, the registration fails.

Since the Linux machine is rebooted, there won't be any connection 
tracking information about the established connections, which is 
required for NAT to work properly.

> Rebooting the Linux machine scatters this behavior among the phones: some are
> randomly registered and some not. Rebooting the phone, and just the phone
> itself, does not change anything.

Hmm... I thought, after rebooting the Linux machine, rebooting the 
problematic phones would help solving the problem. Because, this way the 
phones try to register through a NEW connection (instead of an 
ESTABLISHED one) and the SNAT can be done properly.

Apart from that, just see whether STUN can help to improve your 
situation <http://kb.smartvox.co.uk/voip-sip/sip-devices-nat/>.


Regards,
Vignesh

> Some background I think relevant:
>
> 	1) The Linux ip address is added (one interface, two IPs in two
> 	different nets) further during boot, at rc.local, immediately before
> 	the SNAT rule; No NAT rule was added up to this point.
>
> 	2) if I change the ip address, under the same netmask, of any
> 	non-registered phone, it registers immediately; But this does not assure
> 	it will register again after a new Linux reboot. In fact it may not
> 	register again after that. Already happened.
>
> 	3) All IP-Phones have "keep alive SIP connection" active.
>
> I have a suspicious about what is going on: some race condition.
> But I'd like your thoughts.
>
> Thanx in advance
>
> Regards
>
> Ethy
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: randomly SNATed devices after reboot

[Ethy H. Brito]

> > Investigating with tcpdump I can see, at the external interface, "not
> > snated" packets from those not registered phones. Packets from the other
> > phones are correctly "snatted".
> 
> May be, some phones are trying to register via ESTABLISHED connections 
> which not getting SNATed. So, the registration fails.

That is not exactly the case. They are trying to register via UNREPLIED
connections.

The working phones, they all have ASSURED connections.

> 
> Since the Linux machine is rebooted, there won't be any connection 
> tracking information about the established connections, which is 
> required for NAT to work properly.
> 
> > Rebooting the Linux machine scatters this behavior among the phones: some
> > are randomly registered and some not. Rebooting the phone, and just the
> > phone itself, does not change anything.
> 
> Hmm... I thought, after rebooting the Linux machine, rebooting the 
> problematic phones would help solving the problem. Because, this way the 
> phones try to register through a NEW connection (instead of an 
> ESTABLISHED one) and the SNAT can be done properly.

Ok, these are UDP SIP connections, so "ESTABLISHED" do not apply. I think.

> 
> Apart from that, just see whether STUN can help to improve your 
> situation <http://kb.smartvox.co.uk/voip-sip/sip-devices-nat/>.

STUN won't help.
This is what I think the problem is:
	1) the phones keep sending "keep alive" messages or REGISTER
	requisitions once every few seconds.

	2) during Linux reboot some of these reqs, hit conntrack BEFORE snat is
	alive, creating a wrong tuple at conntrack table.

	3) some phones send requisition AFTER snat is up. And thus I have the
	erratic "registered/not-registered" behavior.

Since conntrack "sees" a packet before snat is in place, it follows bypassing NAT
table with the "wrong" tuple.

What I did to "resolve" this? 
	"conntrack -D" (just after snat rule!)

Question one:
	Is my reasoning correct?
Question two:
	Is this the correct solution? I don't think so.

Any thoughts? Anyone?

Regards

Ethy

 
> 
> Regards,
> Vignesh
> 
> > Some background I think relevant:
> >
> > 	1) The Linux ip address is added (one interface, two IPs in two
> > 	different nets) further during boot, at rc.local, immediately before
> > 	the SNAT rule; No NAT rule was added up to this point.
> >
> > 	2) if I change the ip address, under the same netmask, of any
> > 	non-registered phone, it registers immediately; But this does not
> > assure it will register again after a new Linux reboot. In fact it may not
> > 	register again after that. Already happened.
> >
> > 	3) All IP-Phones have "keep alive SIP connection" active.
> >
> > I have a suspicious about what is going on: some race condition.
> > But I'd like your thoughts.
> >
> > Thanx in advance
> >
> > Regards
> >
> > Ethy
> > --
> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 


-- 

Ethy H. Brito         /"\
InterNexo Ltda.       \ /  CAMPANHA DA FITA ASCII - CONTRA MAIL HTML
+55 (12) 3797-6860     X   ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
S.J.Campos - Brasil   / \ 
 
PGP key: http://www.inexo.com.br/~ethy/0xC3F222A0.asc

Re: randomly SNATed devices after reboot

[Pascal Hambourg]

Hello,

Vigneswaran R a écrit :
> On 05/15/2014 07:12 PM, Ethy H. Brito wrote:
>> Hi All
>>
>> I have this setup in which there are lots of static IPs "SNATed" IP-Phones
>> behind a Linux machine. A very simply NAT machine. Just one SNAT rule for the
>> phones' network.
>>
>> At every Linux machine reboot, some of those phones, randomly, simply does not
>> register at some outside-nat SIP server.
>>
>> Investigating with tcpdump I can see, at the external interface, "not snated"
>> packets from those not registered phones. Packets from the other phones are
>> correctly "snatted".
> 
> May be, some phones are trying to register via ESTABLISHED connections 
> which not getting SNATed. So, the registration fails.

Not ESTABLISHED (that would require return traffic, but existing (NEW).

If a phone sends a SIP packet before the SNAT rule is active, then the
whole SIP flow, including further packets, will not be SNATed until the
related conntrack entry expires. Expiration never happens if the sending
period is shorter than the UDP conntrack expiration delay.

If you don't want this to happen, just DROP all FORWARDed traffic until
the SNAT rule is active.

Re: randomly SNATed devices after reboot

[Ethy H. Brito]

> > 
> > May be, some phones are trying to register via ESTABLISHED connections 
> > which not getting SNATed. So, the registration fails.
> 
> Not ESTABLISHED (that would require return traffic, but existing (NEW).
> 
> If a phone sends a SIP packet before the SNAT rule is active, then the
> whole SIP flow, including further packets, will not be SNATed until the
> related conntrack entry expires. Expiration never happens if the sending
> period is shorter than the UDP conntrack expiration delay.

The phone must send keep-alive in a period shorter than conntrack expiration
period.
If they don't what is the point sending the refresh, right?

> 
> If you don't want this to happen, just DROP all FORWARDed traffic until
> the SNAT rule is active.

<side comment>
Hmmm! I am looking to Jan Engelhardt's Packet Flow picture (2014-Feb-28) and
can not find conntrack in the output path for forwarded packets. I think we
found a glitch in his drawing. Does he read this list?
</side comment>

Nope. I think this is not a ultimate solution because packets still may flow
before FORWARD DROP rule is in place. Your suggestion does not kill the race
condition.
This is what I see, please correct me if I'm wrong:
	1) IP stack is in place during boot 
	2) network parameters are configured (ip addrs, routes, etc)
	3) nf modules are loaded (/etc/modules.d??)
	4) conntrack modules are loaded (also /etc/modules.d)
	5) user scripts are loaded (iptables snat or FORWARD rules included)

If any packets slip between 2 and 5, conntrack already saw the incorrect src
addr.

I need to ensure no packet cross at least before conntrack is loaded, therefore,
before any action I can take via normal boot scripts given the above scenario.

Regards

Ethy

Re: randomly SNATed devices after reboot

[Pascal Hambourg]

Ethy H. Brito a écrit :
> 
>> If you don't want this to happen, just DROP all FORWARDed traffic until
>> the SNAT rule is active.
> 
> <side comment>
> Hmmm! I am looking to Jan Engelhardt's Packet Flow picture (2014-Feb-28) and
> can not find conntrack in the output path for forwarded packets. I think we
> found a glitch in his drawing. Does he read this list?
> </side comment>

The conntrack for forwarded packets is in the PREROUTING path.

> Nope. I think this is not a ultimate solution because packets still may flow
> before FORWARD DROP rule is in place. Your suggestion does not kill the race
> condition.

Well, that's because I put filtering rules in place with default DROP
before enabling the network for obvious safety reasons, and assumed
everyone did the same.

> This is what I see, please correct me if I'm wrong:
> 	1) IP stack is in place during boot 
> 	2) network parameters are configured (ip addrs, routes, etc)
> 	3) nf modules are loaded (/etc/modules.d??)
> 	4) conntrack modules are loaded (also /etc/modules.d)
> 	5) user scripts are loaded (iptables snat or FORWARD rules included)

The order is sysadmin-dependent. You decide. My iptables initscript is
run before the network is configured and activated.

> I need to ensure no packet cross at least before conntrack is loaded

Not necessarily. You're also safe if any forwarded packet is dropped (or
forwarding is disabled) until the SNAT rule is in place. The packets
will be discarded and the conntrack entry will be destroyed immediately.

Re: randomly SNATed devices after reboot

[Sven-Haegar Koch]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 839 bytes --]

On Fri, 16 May 2014, Pascal Hambourg wrote:

> Ethy H. Brito a écrit :
> > This is what I see, please correct me if I'm wrong:
> > 	1) IP stack is in place during boot 
> > 	2) network parameters are configured (ip addrs, routes, etc)
> > 	3) nf modules are loaded (/etc/modules.d??)
> > 	4) conntrack modules are loaded (also /etc/modules.d)
> > 	5) user scripts are loaded (iptables snat or FORWARD rules included)
> 
> The order is sysadmin-dependent. You decide. My iptables initscript is
> run before the network is configured and activated.

And if you are unable for whatever reasons to do it correctly you can 
use the "conntrack" tool after iptables rule setup to flush/delete all 
conntrack connections existing at that time.

c'ya
sven-haegar

-- 
Three may keep a secret, if two of them are dead.
- Ben F.