RE: External IP addresses on internal network

RE: External IP addresses on internal network

[George Vieira]

This could be an internal machine querying aol.com dns server.. which is harmless..

can you show come tcpdumps of this? Was the source IP the aol.com address or was it just the destination.. I would be worried if it was the source IP and it was going out, not in. If it was coming in then don't panic though I woudl prefer to run a caching DNS server locally and block DNS going out.

Thanks,
____________________________________________
George Vieira
Systems Manager
georgev@citadelcomputer.com.au
 

-----Original Message-----
From: Jeffrey Laramie [mailto:JALaramie@Loudoun-Fairfax.com]
Sent: Wednesday, August 27, 2003 1:57 AM
To: netfilter@lists.netfilter.org
Subject: External IP addresses on internal network


Hi all,

I recently updated my configuration by modifying one of my filter INPUT 
rules to specify source IPs coming in from the LAN:

iptables -t filter -A INPUT -p all -i $LAN_Interface -s $LAN_IP_Range -j 
Lan-Host

Packets not meeting this condition (among others) are logged and 
dropped. The intent is to catch any internal packets coming from an 
external IP address. No sooner did I load this rule when I started 
logging packets with an AOL IP address coming from the LAN interface 
going to port 53. I have a small LAN with only a handful of PCs which I 
configured myself, so I'm a bit puzzled.

Have I configured this wrong? Several users connect to AOL through the 
firewall so that's a possible cause, but I don't know why an AOL program 
would spoof IPs. Thoughts?

Re: External IP addresses on internal network

[Jeffrey Laramie]

George Vieira wrote:

>This could be an internal machine querying aol.com dns server.. which is harmless..
>
>can you show come tcpdumps of this? Was the source IP the aol.com address or was it just the destination..
>
George, here is the log entry I got:

Aug 26 15:39:46 NS2 kernel: Filter_INPUT: IN=eth1 OUT= 
MAC=00:c0:f0:69:26:49:52:54:00:de:46:c7:08:00 SRC=172.144.233.136 
DST=192.168.0.24 LEN=73 TOS=0x10 PREC=0x00 TTL=128 ID=1755 PROTO=UDP 
SPT=137 DPT=53 LEN=53

eth1 is the internal LAN interface and 192.168.0.24 is it's IP address. 
It appears from this that I'm getting DNS queries on my internal 
interface from an address that is not in my subnet.

> I would be worried if it was the source IP and it was going out, not in. If it was coming in then don't panic though I woudl prefer to run a caching DNS server locally and block DNS going out.
>  
>
The firewall box is a fully configured DNS server for my public and 
private domains, so getting a DNS request on the internal port is 
normal. I'm just puzzled (and a little concerned) that the client box is 
identified this way.

I have a theory that a laptop on the LAN may have been assigned a valid 
AOL IP during a dialup session and is still configured to use it somehow 
for DNS only (otherwise all the packets would fail this rule). Very odd. 
I'm going to do some more digging on this one.

>Thanks,
>____________________________________________
>George Vieira
>Systems Manager
>georgev@citadelcomputer.com.au
> 
>
>-----Original Message-----
>From: Jeffrey Laramie [mailto:JALaramie@Loudoun-Fairfax.com]
>Sent: Wednesday, August 27, 2003 1:57 AM
>To: netfilter@lists.netfilter.org
>Subject: External IP addresses on internal network
>
>
>Hi all,
>
>I recently updated my configuration by modifying one of my filter INPUT 
>rules to specify source IPs coming in from the LAN:
>
>iptables -t filter -A INPUT -p all -i $LAN_Interface -s $LAN_IP_Range -j 
>Lan-Host
>
>Packets not meeting this condition (among others) are logged and 
>dropped. The intent is to catch any internal packets coming from an 
>external IP address. No sooner did I load this rule when I started 
>logging packets with an AOL IP address coming from the LAN interface 
>going to port 53. I have a small LAN with only a handful of PCs which I 
>configured myself, so I'm a bit puzzled.
>
>Have I configured this wrong? Several users connect to AOL through the 
>firewall so that's a possible cause, but I don't know why an AOL program 
>would spoof IPs. Thoughts?
>
>
>  
>

Re: External IP addresses on internal network

[Jim Carter]

On Tue, 26 Aug 2003, Jeffrey Laramie wrote:
> George, here is the log entry I got:
>
> Aug 26 15:39:46 NS2 kernel: Filter_INPUT: IN=eth1 OUT=
> MAC=00:c0:f0:69:26:49:52:54:00:de:46:c7:08:00 SRC=172.144.233.136
> DST=192.168.0.24 LEN=73 TOS=0x10 PREC=0x00 TTL=128 ID=1755 PROTO=UDP
> SPT=137 DPT=53 LEN=53

It looks to me that 172.144.233.136 is a nameserver, and 192.168.0.24 asked
it for name resolution, and we're looking at its answer.  However, at
present, this machine is refusing connections to port 53, so it's possible
that there used to be a virus on it that tried to use port 53 for some evil
purpose.  Suggestion: do a virus scan on 192.168.0.24.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA  90095-1555
Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: External IP addresses on internal network

[Jeffrey Laramie]

Hi Jim-

Jim Carter wrote:

>On Tue, 26 Aug 2003, Jeffrey Laramie wrote:
>  
>
>>George, here is the log entry I got:
>>
>>Aug 26 15:39:46 NS2 kernel: Filter_INPUT: IN=eth1 OUT=
>>MAC=00:c0:f0:69:26:49:52:54:00:de:46:c7:08:00 SRC=172.144.233.136
>>DST=192.168.0.24 LEN=73 TOS=0x10 PREC=0x00 TTL=128 ID=1755 PROTO=UDP
>>SPT=137 DPT=53 LEN=53
>>    
>>
>
>It looks to me that 172.144.233.136 is a nameserver, and 192.168.0.24 asked
>it for name resolution, and we're looking at its answer.
>
This entry is generated by the built-in filter INPUT chain so I would 
read this as a DNS request coming from 172.144.233.136 (SRC) and 
destined for the host at 192.168.0.24 (which is the firewall's LAN 
facing IP). The firewall host is also a DNS server for my LAN so this 
would be a normal request coming from the LAN **except** for the client 
IP address.

>  However, at
>present, this machine is refusing connections to port 53, so it's possible
>that there used to be a virus on it that tried to use port 53 for some evil
>purpose.  Suggestion: do a virus scan on 192.168.0.24.
>
This is actually pretty interesting (if you don't have a real life). On 
further investigation it appears that these packets only occur when a 
LAN Windows client is connected to AOL. The SRC IPs change for each AOL 
"session" but they are always the IP of an AOL server. It appears the 
AOL server is using the LAN client to query the LAN nameserver and 
signing the packets with its own IP. This seems pretty farfetched but I 
don't have a better explanation.

A point of interest for list members. Most sample scripts and some 
production configurations do little if any filtering of traffic from the 
LAN out to the Net. The theory being that the LAN is trusted. IMHO this 
is a mistake. Malware can get in from e-mail, laptops, wireless 
connections, floppies, etc. and there's no way we can stop all of it. 
IDS is great but pretty pricey for SOHO use. I recently found a worm on 
a client that had gotten in despite our best efforts. The only way I 
knew it was there was by iptables logging (and rejecting) outgoing 
traffic on unauthorized ports. I'll never know if the worm was able to 
find an open port to reach the net, but internally it was contained to 
one box and no harm was done. Just my 2 cents.

Jeff

>James F. Carter          Voice 310 825 2897    FAX 310 206 6673
>UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA  90095-1555
>Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)
>
>  
>

Re: External IP addresses on internal network

[Chris Brenton]

Greets all,

Jeffrey Laramie wrote:
> 
>> On Tue, 26 Aug 2003, Jeffrey Laramie wrote:
>>  
>>
>>> George, here is the log entry I got:
>>>
>>> Aug 26 15:39:46 NS2 kernel: Filter_INPUT: IN=eth1 OUT=
>>> MAC=00:c0:f0:69:26:49:52:54:00:de:46:c7:08:00 SRC=172.144.233.136
>>> DST=192.168.0.24 LEN=73 TOS=0x10 PREC=0x00 TTL=128 ID=1755 PROTO=UDP
>>> SPT=137 DPT=53 LEN=53
>>>   
>>
>>
>> It looks to me that 172.144.233.136 is a nameserver, and 192.168.0.24 
>> asked
>> it for name resolution, and we're looking at its answer.
>>
> This entry is generated by the built-in filter INPUT chain so I would 
> read this as a DNS request coming from 172.144.233.136 (SRC) and 
> destined for the host at 192.168.0.24 (which is the firewall's LAN 
> facing IP).

This obviously is not a legit DNS request because the source port is 
wrong (should be 53 or >1023). My guess is a brain dead Windoze system 
or even more likely, a load balancer.

> The firewall host is also a DNS server for my LAN so this 
> would be a normal request coming from the LAN **except** for the client 
> IP address.

I've seen this before. An internal client goes to access a Web site (say 
www.fubar.org) and the authoritative NS is actually a load balancer. It 
spews suspicious looking traffic at the requesting NS in order to 
generate performance metrics to figure out what IP to serve back 
(assumption being the client is close to the NS).

So if this is the case, you should see a query for a host within the AOL 
domain (owner of the address space) just prior to this traffic.

As for seeing the firewall's private IP in the log entry, are you 
running DNAT on the reply traffic? If so that would explain why it shows 
up as private.

> This is actually pretty interesting (if you don't have a real life).

LOL! :-)

> On 
> further investigation it appears that these packets only occur when a 
> LAN Windows client is connected to AOL. The SRC IPs change for each AOL 
> "session" but they are always the IP of an AOL server. It appears the 
> AOL server is using the LAN client to query the LAN nameserver and 
> signing the packets with its own IP. This seems pretty farfetched but I 
> don't have a better explanation.

I would sniff the internal network and _verify_ the above. My guess is 
its actually coming in from the true IP address, not the internal 
client. Your above log entry shows a TTL of 128 which is normal for a 
Windows box connected to the same LAN as the firewall's internal 
interface. Then again, the source MAC is that of a Cisco router, not a 
PC NIC card.

As mentioned I would verify by capturing the traffic. Something like:

windump -nn -s 1500 -w weird-dns.cap "src port 137 and dst port 53"

from a system inside of your firewall. If you get the log entry but no 
capture, you know its received from outside.

> A point of interest for list members. Most sample scripts and some 
> production configurations do little if any filtering of traffic from the 
> LAN out to the Net. The theory being that the LAN is trusted. IMHO this 
> is a mistake.

I *totally* agree. I write the material and teach SANS perimeter 
security track and one thing I am uber big on is filtering outbound as 
well. Stuff like echo-replies, type 3's, type 11's, NetBIOS/IP, SNMP, 
TFTP, etc. etc. should never be allowed to leave your network. Defense 
in-depth and all of that. If the attacker's stimulus gets in, at least 
you have a shot at blocking the reply.

HTH,
C

Re: External IP addresses on internal network

[Jeffrey Laramie]

Morning all-

>
> I've seen this before. An internal client goes to access a Web site 
> (say www.fubar.org) and the authoritative NS is actually a load 
> balancer. It spews suspicious looking traffic at the requesting NS in 
> order to generate performance metrics to figure out what IP to serve 
> back (assumption being the client is close to the NS).
>
> So if this is the case, you should see a query for a host within the 
> AOL domain (owner of the address space) just prior to this traffic. 

I haven't logged that traffic, but the user is generally checking an AOL 
mail account and could be following links or browsing.

>
>
> As for seeing the firewall's private IP in the log entry, are you 
> running DNAT on the reply traffic? If so that would explain why it 
> shows up as private.

I run SNAT on POSTROUTING to masq the internal LAN but don't do any DNAT.

>
> As mentioned I would verify by capturing the traffic. Something like:
>
> windump -nn -s 1500 -w weird-dns.cap "src port 137 and dst port 53"
>
> from a system inside of your firewall. If you get the log entry but no 
> capture, you know its received from outside. 

I'm a cup of coffee short of a full pot today. I'll have to work on this 
later :-)

>
> I *totally* agree. I write the material and teach SANS perimeter 
> security track and one thing I am uber big on is filtering outbound as 
> well. Stuff like echo-replies, type 3's, type 11's, NetBIOS/IP, SNMP, 
> TFTP, etc. etc. should never be allowed to leave your network. Defense 
> in-depth and all of that. If the attacker's stimulus gets in, at least 
> you have a shot at blocking the reply.
>
Since I'm an IT lightweight, I generally use sample scripts and default 
configs whenever I can. I knew from a philosophical standpoint that I 
wanted outbound filtering. I didn't really know how to do it and I 
didn't find any examples that could help me, so I used the brute force 
method: reject everything and see what it breaks, then open those ports 
only. This method works, sort of, but it can be a problem since many 
programs assign ports dynamically. Anyway, I caught me a big ole worm 
this way so I must be doing something right.

Re: External IP addresses on internal network

[Ramin Dousti]

On Wed, Aug 27, 2003 at 02:50:52PM -0400, Jeffrey Laramie wrote:

> >>Aug 26 15:39:46 NS2 kernel: Filter_INPUT: IN=eth1 OUT=
> >>MAC=00:c0:f0:69:26:49:52:54:00:de:46:c7:08:00 SRC=172.144.233.136
> >>DST=192.168.0.24 LEN=73 TOS=0x10 PREC=0x00 TTL=128 ID=1755 PROTO=UDP
> >>SPT=137 DPT=53 LEN=53
> >
> >It looks to me that 172.144.233.136 is a nameserver, and 192.168.0.24 asked
> >it for name resolution, and we're looking at its answer.
> >
> This entry is generated by the built-in filter INPUT chain so I would 
> read this as a DNS request coming from 172.144.233.136 (SRC) and 
> destined for the host at 192.168.0.24 (which is the firewall's LAN 
> facing IP). The firewall host is also a DNS server for my LAN so this 
> would be a normal request coming from the LAN **except** for the client 
> IP address.

Could it be that the client machine dials up to AOL, receives that IP
address and later it needs to resolve a name and vecause of the DNS
settings on the client machine it tries the query 192.168.0.24 with
its source 172.144.233.136?

Ramin

Re: External IP addresses on internal network

[Jeffrey Laramie]

Hi Ramin,

Ramin Dousti wrote:

>On Wed, Aug 27, 2003 at 02:50:52PM -0400, Jeffrey Laramie wrote:
>
>  
>
>>>>Aug 26 15:39:46 NS2 kernel: Filter_INPUT: IN=eth1 OUT=
>>>>MAC=00:c0:f0:69:26:49:52:54:00:de:46:c7:08:00 SRC=172.144.233.136
>>>>DST=192.168.0.24 LEN=73 TOS=0x10 PREC=0x00 TTL=128 ID=1755 PROTO=UDP
>>>>SPT=137 DPT=53 LEN=53
>>>>        
>>>>
>>>It looks to me that 172.144.233.136 is a nameserver, and 192.168.0.24 asked
>>>it for name resolution, and we're looking at its answer.
>>>
>>>      
>>>
>>This entry is generated by the built-in filter INPUT chain so I would 
>>read this as a DNS request coming from 172.144.233.136 (SRC) and 
>>destined for the host at 192.168.0.24 (which is the firewall's LAN 
>>facing IP). The firewall host is also a DNS server for my LAN so this 
>>would be a normal request coming from the LAN **except** for the client 
>>IP address.
>>    
>>
>
>Could it be that the client machine dials up to AOL, receives that IP
>address and later it needs to resolve a name and vecause of the DNS
>settings on the client machine it tries the query 192.168.0.24 with
>its source 172.144.233.136?
>
Thanks for the suggestion. That was my first thought since one of the 
LAN clients is a notebook with dialup ability, but with a DSL connection 
through the LAN it's not used now. I did check it though to see if it 
still had an AOL IP assigned to it or an AOL server listed for DNS. It 
didn't, and the log timestamp indicates that these packets are occuring 
when a different client (with no dialup) is checking AOL mail.

Re: External IP addresses on internal network

[Ramin Dousti]

On Thu, Aug 28, 2003 at 12:05:24PM -0400, Jeffrey Laramie wrote:
> 
> >
> >Could it be that the client machine dials up to AOL, receives that IP
> >address and later it needs to resolve a name and vecause of the DNS
> >settings on the client machine it tries the query 192.168.0.24 with
> >its source 172.144.233.136?
> >
> Thanks for the suggestion. That was my first thought since one of the LAN
> clients is a notebook with dialup ability, but with a DSL connection through
> the LAN it's not used now. I did check it though to see if it still had an
> AOL IP assigned to it or an AOL server listed for DNS. It didn't, and the log
> timestamp indicates that these packets are occuring when a different client
> (with no dialup) is checking AOL mail.
> 

What about this theory (although I don't know anything about the AOL stuff)
that the client connects to AOL. AOL sets up a tunnel with this client
and assigns 172.144.233.136 to it. Then due to the static DNS settings
on the client, a DNS query is made to your named on the firewall, instead
of using AOL's DNS server?

Ramin

Re: External IP addresses on internal network

[Jeffrey Laramie]

Hey all,

You guys must hate me by now. I've had a handful of good responses and 
been responding and cc'ing the list trying to keep everyone current, but 
with the list lag it isn't working very well. If your box isn't full 
yet, it will be. Sorry.

Ramin Dousti wrote:

>On Thu, Aug 28, 2003 at 12:05:24PM -0400, Jeffrey Laramie wrote:
>  
>
>>>Could it be that the client machine dials up to AOL, receives that IP
>>>address and later it needs to resolve a name and vecause of the DNS
>>>settings on the client machine it tries the query 192.168.0.24 with
>>>its source 172.144.233.136?
>>>
>>>      
>>>
>>Thanks for the suggestion. That was my first thought since one of the LAN
>>clients is a notebook with dialup ability, but with a DSL connection through
>>the LAN it's not used now. I did check it though to see if it still had an
>>AOL IP assigned to it or an AOL server listed for DNS. It didn't, and the log
>>timestamp indicates that these packets are occuring when a different client
>>(with no dialup) is checking AOL mail.
>>
>>    
>>
>
>What about this theory (although I don't know anything about the AOL stuff)
>that the client connects to AOL. AOL sets up a tunnel with this client
>and assigns 172.144.233.136 to it. Then due to the static DNS settings
>on the client, a DNS query is made to your named on the firewall, instead
>of using AOL's DNS server?
>
I don't know much about tunneling, but this sounds possible. Almost like 
a remote shell where the server session runs on client hardware, except 
here the DNS call gets mis-handled and sent to the client nameserver 
instead?

Chris had a very good theory too:  (quoted here in case you didn't get 
it yet)

> This obviously is not a legit DNS request because the source port is 
> wrong (should be 53 or >1023). My guess is a brain dead Windoze system 
> or even more likely, a load balancer.
>
>> The firewall host is also a DNS server for my LAN so this would be a 
>> normal request coming from the LAN **except** for the client IP address.
>
>
> I've seen this before. An internal client goes to access a Web site 
> (say www.fubar.org) and the authoritative NS is actually a load 
> balancer. It spews suspicious looking traffic at the requesting NS in 
> order to generate performance metrics to figure out what IP to serve 
> back (assumption being the client is close to the NS).
>
> So if this is the case, you should see a query for a host within the 
> AOL domain (owner of the address space) just prior to this traffic.

This is good stuff, thanks guys. I'll let you know if I find out 
anything definative.

Jeff

Re: External IP addresses on internal network

[Michael J. Tubby B.Sc. (Hons) G8TIC]

> 
> A point of interest for list members. Most sample scripts and some 
> production configurations do little if any filtering of traffic from the 
> LAN out to the Net. The theory being that the LAN is trusted. IMHO this 
> is a mistake. Malware can get in from e-mail, laptops, wireless 
> connections, floppies, etc. and there's no way we can stop all of it. 
> IDS is great but pretty pricey for SOHO use. I recently found a worm on 
> a client that had gotten in despite our best efforts. The only way I 
> knew it was there was by iptables logging (and rejecting) outgoing 
> traffic on unauthorized ports. I'll never know if the worm was able to 
> find an open port to reach the net, but internally it was contained to 
> one box and no harm was done. Just my 2 cents.
> 
> Jeff
> 

I filter outbound stuff in the OUTPUT table (packets from my firewall
host to the net) and the FORWARD table.

What you should be filtering is packet egress, that is your 192.168 or
whatever internal LAN leaking 'naked' (un-NATted) to the internet at
large.

I do this in two places, on the main firewall box between out "inside"
and "outside" in iptables and then again at the Cisco router facing
PSInet with an ACL that drop & log any packets from anything other
than our PSInet Class-C, so there's no way we can "leak" on to the 'net.

Doesn't everyone do this?

Mike

Re: External IP addresses on internal network

[Jeffrey Laramie]

Hey Mike,

Michael J. Tubby B.Sc. (Hons) G8TIC wrote:

>What you should be filtering is packet egress, that is your 192.168 or
>whatever internal LAN leaking 'naked' (un-NATted) to the internet at
>large.
>  
>
Yes, I do this. I also filter coming from the LAN on the INPUT table 
since I consider any box a potential risk.

>I do this in two places, on the main firewall box between out "inside"
>and "outside" in iptables and then again at the Cisco router facing
>PSInet with an ACL that drop & log any packets from anything other
>than our PSInet Class-C, so there's no way we can "leak" on to the 'net.
>  
>
My DSL provider owns and maintains my router. I never thought about 
filtering at the router. Hmm, I'll have to ask about that.

Jeff

External IP addresses on internal network

[Jeffrey Laramie]

Hi all,

I recently updated my configuration by modifying one of my filter INPUT 
rules to specify source IPs coming in from the LAN:

iptables -t filter -A INPUT -p all -i $LAN_Interface -s $LAN_IP_Range -j 
Lan-Host

Packets not meeting this condition (among others) are logged and 
dropped. The intent is to catch any internal packets coming from an 
external IP address. No sooner did I load this rule when I started 
logging packets with an AOL IP address coming from the LAN interface 
going to port 53. I have a small LAN with only a handful of PCs which I 
configured myself, so I'm a bit puzzled.

Have I configured this wrong? Several users connect to AOL through the 
firewall so that's a possible cause, but I don't know why an AOL program 
would spoof IPs. Thoughts?