Leaking ICMP and UDP packets

Leaking ICMP and UDP packets

[Daniel David Benson]

I essentially have everything go out as a global nat, but every now and 
then a tcpdump on the frontside interface is showing some ICMP and UDP 
packets not getting natted.  Anyone ever seen this before?  It's not a 
major deal as we are are having our front edge router handle these ugly 
packets, but I'd like to tighten it up.


Thanks!

-Dan

Re: Leaking ICMP and UDP packets

[Frank Gruellich]

* Daniel David Benson <dan@spoontail.com> 19. Apr 04:
> I essentially have everything go out as a global nat,

What exactly is you rule?

> but every now and then a tcpdump on the frontside interface is showing
> some ICMP and UDP packets not getting natted.

What exactly do you tcpdump?  (Cmdline, please.)  What exactly is the
output of it?

> Anyone ever seen this before?

No.

> It's not a major deal as we are are having our front edge router
> handle these ugly packets, but I'd like to tighten it up.

I am just curious: does it NAT, too?  (This has nothing to do with your
problem, whatever it may be.)

 Regards, Frank.
-- 
Sigmentation fault

Re: Leaking ICMP and UDP packets

[Daniel David Benson]

Here's the catchall rule:

$EXT_IFACE = eth0
$EXT_IP = Some Internet Routable Public IP.

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE -j SNAT --to-source $EXT_IP

Here's what I am doing with tcpdump (eth0 is the public interface):

tcpdump -vvv -i eth0 net 10.0.0.0/8

Unfortunately I can't send any of the packets captured via tcpdump....as 
I get about 5-10 a day and I killed the screen that had last weeks packets 
-- I'm recording them better now :)

-Dan


On Mon, 19 Apr 2004, Frank Gruellich wrote:

> * Daniel David Benson <dan@spoontail.com> 19. Apr 04:
> > I essentially have everything go out as a global nat,
> 
> What exactly is you rule?
> 
> > but every now and then a tcpdump on the frontside interface is showing
> > some ICMP and UDP packets not getting natted.
> 
> What exactly do you tcpdump?  (Cmdline, please.)  What exactly is the
> output of it?
> 
> > Anyone ever seen this before?
> 
> No.
> 
> > It's not a major deal as we are are having our front edge router
> > handle these ugly packets, but I'd like to tighten it up.
> 
> I am just curious: does it NAT, too?  (This has nothing to do with your
> problem, whatever it may be.)
> 
>  Regards, Frank.
> 

Re: Leaking ICMP and UDP packets

[Daniel David Benson]

> > I am just curious: does it NAT, too?  (This has nothing to do with your
> > problem, whatever it may be.)

Overlooked this question.  No, the edge router does no natting -- just 
routes.  We block private IPs going in/out the edge router -- that's how 
we handle them, and how we found these packets.  By the way, here's some 
needed version numbers:

iptables: 1.2.8
kernel: 2.4.20-28.8smp (yeah I know -- moving away from RH8 soon)

Outside of these packets, NAT is working fine -- in fact, UDP and ICMP are 
being natted correctly outside of these packets.

Thanks!

-Dan

Re: Leaking ICMP and UDP packets

[Antony Stone]

On Monday 19 April 2004 5:17 pm, Daniel David Benson wrote:

> I essentially have everything go out as a global nat, but every now and
> then a tcpdump on the frontside interface is showing some ICMP and UDP
> packets not getting natted.

"Some"?   I agree with Frank - let's see an example, please.

> Anyone ever seen this before?

No.   As far as I'm concerned, if you tell netfilter to NAT, then it NATs.

> It's not a major deal as we are are having our front edge router handle
> these ugly packets, but I'd like to tighten it up.

Are you sure they are un-NATted packets "leaking" through netfilter, and not 
ICMP packets being sent back to some remote system by the firewall itself, 
saying "host unreachable" etc?

Other information which may be useful to diagnose this problem:

1. What's your Internet bandwidth?
2. How many conntrack table entries do you typically have (wc -l 
/proc/net/ip_conntrack)?
3. Do you have any DNAT rules for incoming packets, as well as the SNAT rule 
you've already posted?
4. Are you doing anything in the mangle table?
5. Do you run any services on the firewall, or is it just a filtering router?

Regards,

Antony.

-- 
Anyone that's normal doesn't really achieve much.

 - Mark Blair, Australian rocket engineer

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Leaking ICMP and UDP packets

[Daniel David Benson]

> Are you sure they are un-NATted packets "leaking" through netfilter, and not 
> ICMP packets being sent back to some remote system by the firewall itself, 
> saying "host unreachable" etc?

Finally, after 24 hours I got the following packets on my front edge 
(public) interface:

09:57:48.388232 10.101.10.103 > 169.254.85.249: icmp: time exceeded in-transit
09:58:03.386143 10.101.10.103 > 169.254.85.249: icmp: time exceeded in-transit
09:58:18.386973 10.101.10.103 > 169.254.85.249: icmp: time exceeded in-transit

* This internal IP isn't any of the 1 to 1 NATs listed below and should 
come out as the global NAT IP.

> 1. What's your Internet bandwidth?

It's a 6 Mb/s pipe -- symetric -- ethernet drop within from a provider 
within our building.

> 2. How many conntrack table entries do you typically have (wc -l 
> /proc/net/ip_conntrack)?

Looks to hover around 4300 entries.  This is what I set the table size to:

echo "200000" > /proc/sys/net/ipv4/ip_conntrack_max

> 3. Do you have any DNAT rules for incoming packets, as well as the SNAT rule 
> you've already posted?

Here are all my NAT entries:

####
####  PREROUTING Chain
####

$IPTABLES -t nat -A PREROUTING -i $EXT_IFACE \
-d $EXT_DMZ1 -j DNAT --to-destination $INT_DMZ1
 
$IPTABLES -t nat -A PREROUTING -i $EXT_IFACE \
-d $EXT_DMZ2 -j DNAT --to-destination $INT_DMZ2
 
$IPTABLES -t nat -A PREROUTING -i $EXT_IFACE \
-d $EXT_INEWS -j DNAT --to-destination $INT_INEWS
                                                                                
$IPTABLES -t nat -A PREROUTING -i $EXT_IFACE \
-d $EXT_CABLE -j DNAT --to-destination $INT_CABLE
                                                                                
$IPTABLES -t nat -A PREROUTING -i $EXT_IFACE \
-d $EXT_CABLEDEV -j DNAT --to-destination $INT_CABLEDEV
                                                                                
$IPTABLES -t nat -A PREROUTING -i $EXT_IFACE \
-d $EXT_TRANSFER -j DNAT --to-destination $INT_TRANSFER
                                                                                
$IPTABLES -t nat -A PREROUTING -i eth0 \
-d $EXT_QADMZ1 -j DNAT --to-destination $INT_QADMZ1
                                                                                
####
####  POSTROUTING Chain
####

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_DMZ1 -j SNAT --to-source $EXT_DMZ1

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_DMZ2 -j SNAT --to-source $EXT_DMZ2

#$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
#-s $INT_QAWEB1 -j SNAT --to-source $EXT_QAWEB1

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_INEWS -j SNAT --to-source $EXT_INEWS

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_CABLE -j SNAT --to-source $EXT_CABLE

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_CABLEDEV -j SNAT --to-source $EXT_CABLEDEV

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_TRANSFER -j SNAT --to-source $EXT_TRANSFER

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE \
-s $INT_QADMZ1 -j SNAT --to-source $EXT_QADMZ1

$IPTABLES -t nat -A POSTROUTING -o $EXT_IFACE -j SNAT --to-source $EXT_IP

> 4. Are you doing anything in the mangle table?

No

> 5. Do you run any services on the firewall, or is it just a filtering router?

It's just a filtering router.  I have three NICS -- Public, Private, and 
DMZ.  Everything seems to NAT fine except some of these ICMP type packets 
and it looks like UDP packets associated with NTP.

Any help would be much appreciated.  Sorry for such a vague initial 
writeup.

-Dan

Re: Leaking ICMP and UDP packets

[Antony Stone]

On Tuesday 20 April 2004 6:13 pm, Daniel David Benson wrote:

> > Are you sure they are un-NATted packets "leaking" through netfilter, and
> > not ICMP packets being sent back to some remote system by the firewall
> > itself, saying "host unreachable" etc?
>
> Finally, after 24 hours I got the following packets on my front edge
> (public) interface:
>
> 09:57:48.388232 10.101.10.103 > 169.254.85.249: icmp: time exceeded
> in-transit 09:58:03.386143 10.101.10.103 > 169.254.85.249: icmp: time
> exceeded in-transit 09:58:18.386973 10.101.10.103 > 169.254.85.249: icmp:
> time exceeded in-transit
>
> * This internal IP isn't any of the 1 to 1 NATs listed below and should
> come out as the global NAT IP.

Correct me if I'm wrong, but isn't 169.254.85.249 one of those Microsoft 
"let's make up a network range if we can't find anyone to talk to" addresses?

In other words, the original packet did not come from the outside world, 
therefore did not come through your firewall (it came from a lonely Windows 
machine inside your network which has decided to create its own IP address, 
and has then decided to send packets to 10.101.10.103, which returned an ICMP 
error through your default gateway), and therefore there was no automatic 
reverse NAT in place to set the source address to what you would expect?

I bet you can't find the original packet, to which this ICMP TTL exceeded 
message corresponds, in your packet sniffer logs, because it never came in 
through the firewall in the first place.

Regards,

Antony.

-- 
This email is intended for the use of the individual addressee(s) named above 
and may contain information that is confidential, privileged or unsuitable 
for overly sensitive persons with low self-esteem, no sense of humour, or 
irrational religious beliefs.

If you have received this email in error, you are required to shred it 
immediately, add some nutmeg, three egg whites and a dessertspoonful of 
caster sugar.   Whisk until soft peaks form, then place in a warm oven for 40 
minutes.   Remove promptly and let stand for 2 hours before adding some 
decorative kiwi fruit and cream.   Then notify me immediately by return email 
and eat the original message.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Leaking ICMP and UDP packets

[Daniel David Benson]

Ok, so if I understand correctly, the 10.101.10.103 host is trying to 
respond to this windows box -- it only knows the default gateway to get to 
that 169 address.  It doesn't NAT because it doesn't have a corresponding 
"initiated" NAT entry?  I guess I'm confused as to why it doesn't match 
the global NAT entry -- I suspect it's my lack of understanding of how NAT 
is implemented.  In the end, I should be blocking these 169 requests in 
the FORWARD table anyway.  Is it common practice to set thet NAT chains 
default policies to DROP?  I guess, I'd like to know the best way to stop 
these bad boys from getting out to be a good netizen.

-Dan



On Tue, 20 Apr 2004, Antony Stone wrote:

> On Tuesday 20 April 2004 6:13 pm, Daniel David Benson wrote:
> 
> > > Are you sure they are un-NATted packets "leaking" through netfilter, and
> > > not ICMP packets being sent back to some remote system by the firewall
> > > itself, saying "host unreachable" etc?
> >
> > Finally, after 24 hours I got the following packets on my front edge
> > (public) interface:
> >
> > 09:57:48.388232 10.101.10.103 > 169.254.85.249: icmp: time exceeded
> > in-transit 09:58:03.386143 10.101.10.103 > 169.254.85.249: icmp: time
> > exceeded in-transit 09:58:18.386973 10.101.10.103 > 169.254.85.249: icmp:
> > time exceeded in-transit
> >
> > * This internal IP isn't any of the 1 to 1 NATs listed below and should
> > come out as the global NAT IP.
> 
> Correct me if I'm wrong, but isn't 169.254.85.249 one of those Microsoft 
> "let's make up a network range if we can't find anyone to talk to" addresses?
> 
> In other words, the original packet did not come from the outside world, 
> therefore did not come through your firewall (it came from a lonely Windows 
> machine inside your network which has decided to create its own IP address, 
> and has then decided to send packets to 10.101.10.103, which returned an ICMP 
> error through your default gateway), and therefore there was no automatic 
> reverse NAT in place to set the source address to what you would expect?
> 
> I bet you can't find the original packet, to which this ICMP TTL exceeded 
> message corresponds, in your packet sniffer logs, because it never came in 
> through the firewall in the first place.
> 
> Regards,
> 
> Antony.
> 
> 

Re: Leaking ICMP and UDP packets

[Antony Stone]

On Tuesday 20 April 2004 7:06 pm, Daniel David Benson wrote:

> Ok, so if I understand correctly, the 10.101.10.103 host is trying to
> respond to this windows box -- it only knows the default gateway to get to
> that 169 address.

Yes, and the default gateway (your firewall) only knows its external interface 
to send out packets which don't belong to the internal LAN/s.

>  It doesn't NAT because it doesn't have a corresponding
> "initiated" NAT entry?  I guess I'm confused as to why it doesn't match
> the global NAT entry -- I suspect it's my lack of understanding of how NAT
> is implemented.

Hm, actually, yes, you have a point there - you do have a generic SNAT rule 
which should apply to any packets leaving the external interface.   I don't 
have an answer for that one right now.

>  In the end, I should be blocking these 169 requests in
> the FORWARD table anyway.

Yes, that would be a good plan.

> Is it common practice to set the NAT chains default policies to DROP?

NO!!!!   *Never* set a nat table (or a mangle table) to have any default 
policy other than ACCEPT.

Think really *really* hard before using a rule with a DROP target in the nat 
or mangle tables (it can be valid sometimes, but make sure know what it will 
do before you use it).

>  I guess, I'd like to know the best way to stop these bad boys from getting
> out to be a good netizen.

Well, I wouldn't worry about it too much - an upstream ISP will drop them soon 
enough with that destination address.

If you do want to drop them yourself, however, start with the addresses listed 
in http://www.rfc-editor.org/rfc/rfc3330.txt and block anything entering or 
leaving your network with a source or destination address in these ranges.

Regards,

Antony.

-- 
"The future is already here.   It's just not evenly distributed yet."

 - William Gibson

                                                     Please reply to the list;
                                                           please don't CC me.