incoming interface confusion question

incoming interface confusion question

[Shaun T. Erickson]

Are there any cases where iptables can be confused about what interface 
a packet came in on? Can a packet arriving on interface A ever be 
reported as arriving on interface B?

I had an incident this weekend, and am trying to be certain that the 
packets came in the interface my system said it did. It's a Red Hat 9 
system, running their stock 2.4.20-8 kernel.

	-ste

Re: incoming interface confusion question

[Shaun T. Erickson]

Shaun T. Erickson wrote:

> Are there any cases where iptables can be confused about what interface 
> a packet came in on? Can a packet arriving on interface A ever be 
> reported as arriving on interface B?
> 
> I had an incident this weekend, and am trying to be certain that the 
> packets came in the interface my system said it did. It's a Red Hat 9 
> system, running their stock 2.4.20-8 kernel.

Please, can anyone answer this for me? I'm trying to prove or disprove a 
theory that would explain an apparent intrusion incident over the 
weekend. It's very important that I know the definitive answer to this. 
Thanks.

	-ste

Re: incoming interface confusion question

[John A. Sullivan III]

On Mon, 2004-06-21 at 15:28, Shaun T. Erickson wrote:
> Shaun T. Erickson wrote:
> 
> > Are there any cases where iptables can be confused about what interface 
> > a packet came in on? Can a packet arriving on interface A ever be 
> > reported as arriving on interface B?
> > 
> > I had an incident this weekend, and am trying to be certain that the 
> > packets came in the interface my system said it did. It's a Red Hat 9 
> > system, running their stock 2.4.20-8 kernel.
> 
> Please, can anyone answer this for me? I'm trying to prove or disprove a 
> theory that would explain an apparent intrusion incident over the 
> weekend. It's very important that I know the definitive answer to this. 
> Thanks.
> 
> 	-ste
I have never experienced it but I'm sure there are others more
experienced than I on this list - John
-- 
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net 

Re: incoming interface confusion question [LONG]

[Shaun T. Erickson]

Ok, here's some more detail on what happened.

I have a SonicWall firewall as my external firewall. Its external 
address and the address of the systems in the DMZ hanging off of it, are 
in a pool of 16 real IPs we got from the ISP.

The LAN port of the SonicWall is connected to a hub. The only other 
thing connected to the hub is eth0 of my internal iptables 
router/firewall. I have an internal lan hanging off eth1 of the iptable 
system, and a second internal lan hanging off eth2 of the iptables 
system. The SonicWall has two static routes pointing traffic for my 
internal lans to eth0 on the iptables system. Both of my lans use 
non-routable private ip address space.

What happened is that the iptables system dropped what appear to be 
quite methodical scans of system on one of my lans by sites out on the 
internet. It saw the packets coming in eth0 and destined to head out 
eth1 and, based on my rulesets, logged and dropped them.

The problem is, the iptables system should have never seen them in the 
first place. The source addresses are from multiple internet sites, 
coming from either port 80 or port 443, and were destined for the 
private address space of one of my internal lans. The initial 
destination port seemed randomly chosen, but then incremented by 1. Most 
often, 92 bytes was sent to each port, but not always. The ruleset of 
the SonicWall should not allow such traffic through it. They looked over 
my ruleset and could find nothing that would allow such traffic to pass 
their device. Moreover, traffic destined for private non-routable 
address space can't be routed over the internet, as routers along the 
way would drop it, so it would seem the traffic couldn't have come in 
the wan port of the SonicWall. Now, maybe they broke into a system on my 
DMZ. Well, I did an nmap scan of my internal lan from the DMZ and the 
sonicwall screamed in it's logs that a port scan was happening and it 
dropped the packets. The iptables system never saw the scan. So that 
suggests that it didn't come from my DMZ, either.

How else could the traffic have arrived? Well, outside the sonicwall, 
there is a switch between it and my external router. Plugged into that 
switch is a wireless access point. If someone was on that, the packets 
would still have to come through the sonicwall to be seen by the 
iptables system. Sonicwall says the packets could not have come through 
their device.

Ok. Next theory: some system on the inside is compromised and spoofing 
the source address of the packets it sends out. So maybe something on 
lan2 tried to scan something on lan1. The problem with that is that on 
the iptables system, I have a rule that says if a packet comes in eth2 
with a source address other than one from the network attached to that 
interface, the system is to drop the packet. I have a similar rule for 
packets coming in eth1 from the other lan. So, that suggests that the 
packets couldn't have come from the inside, either.

We have a separate, smaller sonicwall that acts as our vpn concentrator. 
Only vpn traffic passes through that device. If someone broke in via our 
vpn, they'd either get a dhcp address on that lan or they could pretend 
to be something else. If they spoofed packet source from there, again 
the rule saying drop packets with a source other than that net would 
kick in.

So, The ruleset on the sonicwall says it couldn't have come from the 
outside, and the rulset on the iptables system says it couldn't have 
come from the inside.

Now what?

	-ste

Re: incoming interface confusion question

[Antony Stone]

On Monday 21 June 2004 10:13 pm, Shaun T. Erickson wrote:

> Ok, here's some more detail on what happened.
>
> I have a SonicWall firewall as my external firewall. Its external
> address and the address of the systems in the DMZ hanging off of it, are
> in a pool of 16 real IPs we got from the ISP.
>
> The LAN port of the SonicWall is connected to a hub. The only other
> thing connected to the hub is eth0 of my internal iptables
> router/firewall. I have an internal lan hanging off eth1 of the iptable
> system, and a second internal lan hanging off eth2 of the iptables
> system. The SonicWall has two static routes pointing traffic for my
> internal lans to eth0 on the iptables system. Both of my lans use
> non-routable private ip address space.

Since you use non-routable addresses on your internal LANs, and you say that 
the SonicWall has static routes pointing to those network ranges, I assume 
you are doing the NAT (which is necessary for clients to be able to access 
the Internet) on the SonicWall, not on the netfilter machine.

Unless you have any DNAT rules on your SonicWall redirecting some public 
address/es to your private address space, I think you can rule out someone on 
the Internet having sent the packets in to your internal address space, 
because they simply wouldn't be able to get the packets through to private 
addresses.

> What happened is that the iptables system dropped what appear to be
> quite methodical scans of system on one of my lans by sites out on the
> internet. It saw the packets coming in eth0 and destined to head out
> eth1 and, based on my rulesets, logged and dropped them.
>
> The source addresses are from multiple internet sites,
> coming from either port 80 or port 443, and were destined for the
> private address space of one of my internal lans. The initial
> destination port seemed randomly chosen, but then incremented by 1. Most
> often, 92 bytes was sent to each port, but not always.

What sort of packets were they?   You've said they were from source ports 80 
and 443, so I assume we're talking TCP, not UDP.   What flags were set?   
SYN? ACK? FIN? RST? (URG & PSH aren't very interesting really)

Packets from source ports 80 and 443 are to be expected - as replies from a 
web server - so I wonder whether you should be investigating some machine on 
your internal network (rather than puzzling over your firewall), to see 
whether it was sending *out* packets, which you have simply seen the replies 
to?

A Windows client which makes repeated connections to a web server will very 
likely choose successively higher source port numbers (which are the 
destination port numbers in the reply packets) for each connection attempt, 
therefore several things about what you've said suggest to me that you're 
seeing reply packets, not an externally-initiated port scan.

The fact that the packets you've seen are to multiple addresses within your 
own LAN in fact makes me wonder whether someone inside your network has 
(deliberately or unwittingly) performed a TCP port 80 scan of several 
external addresses, using source address spoofing to disguise the address of 
the machine doing the scan?

It's not unreasonable to think that a worm or trojan infection of an internal 
system could cause this.

Maybe it would be worth adding an outbound logging rule (perhaps just for port 
443 to keep the signal:noise ratio higher than it would be on port 80) to see 
if something similar happens again, and see whether you had outbound packets 
preceding the inbound ones?

Regards,

Antony.

-- 
Software development can be quick, high quality, or low cost.

The customer gets to pick any two out of three.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: incoming interface confusion question

[Shaun T. Erickson]

Antony Stone wrote:

> Since you use non-routable addresses on your internal LANs, and you say that 
> the SonicWall has static routes pointing to those network ranges, I assume 
> you are doing the NAT (which is necessary for clients to be able to access 
> the Internet) on the SonicWall, not on the netfilter machine.

Correct. (Btw, if I come accross as niaeve, I apologize. I've never had 
to deal with a potential incident before, so I'll admit to being a bit 
spooked at the moment. I just replaced the previous admin and am still 
coming up to speed. Please forgive me.)

> Unless you have any DNAT rules on your SonicWall redirecting some public 
> address/es to your private address space, I think you can rule out someone on 
> the Internet having sent the packets in to your internal address space, 
> because they simply wouldn't be able to get the packets through to private 
> addresses.

There are no rules on the sonicwall that allow traffic initiated on the 
wan to go to the lan. There is one rule that allows one specific system 
in the dmz to connect to one specific system on an internal lan, on it's 
MySQL port.

>>The source addresses are from multiple internet sites,
>>coming from either port 80 or port 443, and were destined for the
>>private address space of one of my internal lans. The initial
>>destination port seemed randomly chosen, but then incremented by 1. Most
>>often, 92 bytes was sent to each port, but not always.
> 
> 
> What sort of packets were they?   You've said they were from source ports 80 
> and 443, so I assume we're talking TCP, not UDP.   What flags were set?   
> SYN? ACK? FIN? RST? (URG & PSH aren't very interesting really)

They are all TCP packets. The breakdown of flags was:

     140 ACK FIN URGP=0
     202 ACK PSH FIN URGP=0
     510 ACK PSH URGP=0
       2 ACK RST URGP=0
     350 ACK SYN URGP=0
   14719 ACK URGP=0
       3 RST URGP=0

> Packets from source ports 80 and 443 are to be expected - as replies from a 
> web server - so I wonder whether you should be investigating some machine on 
> your internal network (rather than puzzling over your firewall), to see 
> whether it was sending *out* packets, which you have simply seen the replies 
> to?

Perhaps.

> A Windows client which makes repeated connections to a web server will very 
> likely choose successively higher source port numbers (which are the 
> destination port numbers in the reply packets) for each connection attempt, 
> therefore several things about what you've said suggest to me that you're 
> seeing reply packets, not an externally-initiated port scan.

I don't fully understand this. If a Windows client on my lan makes 
repeated attempts to connect to web sites on the internet, it sends each 
request from a successively higher port number each time, rather than 
from the same port each time? And the port that it requests from then 
becomes the destination port for the return packet?

> The fact that the packets you've seen are to multiple addresses within your 
> own LAN in fact makes me wonder whether someone inside your network has 
> (deliberately or unwittingly) performed a TCP port 80 scan of several 
> external addresses, using source address spoofing to disguise the address of 
> the machine doing the scan?
> 
> It's not unreasonable to think that a worm or trojan infection of an internal 
> system could cause this.

Ok. I'm willing to buy that, except for one thing. Eth2 of the netfilter 
box creates the 192.168.32.0/24 net. Eth1 of the netfilter box creates 
the 192.168.40.0/24 net. My firewall contains (among others) these rules:

$IPTABLES -A INPUT   -i $DB_IFACE -s ! $DB_ADDRESSES -j DROP
$IPTABLES -A FORWARD -i $DB_IFACE -s ! $DB_ADDRESSES -j DROP
$IPTABLES -A INPUT   -i $DEV_IFACE -s ! $DEV_ADDRESSES -j DROP
$IPTABLES -A FORWARD -i $DEV_IFACE -s ! $DEV_ADDRESSES -j DROP

where:

DB_IFACE=eth1
DEV_IFACE=eth2
DB_ADDRESSES=192.168.40.0/24
DEV_ADDRESSES=192.168.32.0/24

Wouldn't those rules cause any packets coming from my internal lans, 
with spoofed source addresses, to be dropped? If so, then they couldn't 
have contacted internet websites for me to then see reply packets from, 
right?

> Maybe it would be worth adding an outbound logging rule (perhaps just for port 
> 443 to keep the signal:noise ratio higher than it would be on port 80) to see 
> if something similar happens again, and see whether you had outbound packets 
> preceding the inbound ones?

I shall consider it.

	-ste

[SOLVED] Re: incoming interface confusion question

[Shaun T. Erickson]

Thanks for your comments and questions, everyone - they got me onto a 
different train of thought which quickly led me to a resolution - I'm in 
your debt. :)

The traffic was indeed response packets to connections made from systems 
on my lan. It seems that I put in a rule on my netfilter box, Friday, 
that allowed out some traffic that had been bottled up, just waiting to 
get to the internet. This traffic turned out to be windows servers 
looking for updates from microsoft. The firewall did let the return 
packets back, but logged them as if it hadn't. The log rule was supposed 
to log anything that was about to hit the default chain policy of drop, 
but the rule I added Friday got added after the logging rule, instead of 
before it. So, it was logged, then accepted.

Mystery solved.

Now, if I were a networking guy, instead of a sysadmin, or at least one 
with more networking knowledge, I'd've figured this out this morning, 
and saved myself a day's wild goose chase, and the additional gray 
hairs. Sigh.

Again, thanks.

	-ste

Re: incoming interface confusion question [LONG]

[John A. Sullivan III]

On Mon, 2004-06-21 at 17:13, Shaun T. Erickson wrote:
> Ok, here's some more detail on what happened.
> 
> I have a SonicWall firewall as my external firewall. Its external 
> address and the address of the systems in the DMZ hanging off of it, are 
> in a pool of 16 real IPs we got from the ISP.
> 
> The LAN port of the SonicWall is connected to a hub. The only other 
> thing connected to the hub is eth0 of my internal iptables 
> router/firewall. I have an internal lan hanging off eth1 of the iptable 
> system, and a second internal lan hanging off eth2 of the iptables 
> system. The SonicWall has two static routes pointing traffic for my 
> internal lans to eth0 on the iptables system. Both of my lans use 
> non-routable private ip address space.
> 
> What happened is that the iptables system dropped what appear to be 
> quite methodical scans of system on one of my lans by sites out on the 
> internet. It saw the packets coming in eth0 and destined to head out 
> eth1 and, based on my rulesets, logged and dropped them.
> 
> The problem is, the iptables system should have never seen them in the 
> first place. The source addresses are from multiple internet sites, 
> coming from either port 80 or port 443, and were destined for the 
> private address space of one of my internal lans. The initial 
> destination port seemed randomly chosen, but then incremented by 1. Most 
> often, 92 bytes was sent to each port, but not always. The ruleset of 
> the SonicWall should not allow such traffic through it. They looked over 
> my ruleset and could find nothing that would allow such traffic to pass 
> their device. Moreover, traffic destined for private non-routable 
> address space can't be routed over the internet, as routers along the 
> way would drop it, so it would seem the traffic couldn't have come in 
> the wan port of the SonicWall. Now, maybe they broke into a system on my 
> DMZ. Well, I did an nmap scan of my internal lan from the DMZ and the 
> sonicwall screamed in it's logs that a port scan was happening and it 
> dropped the packets. The iptables system never saw the scan. So that 
> suggests that it didn't come from my DMZ, either.
> 
> How else could the traffic have arrived? Well, outside the sonicwall, 
> there is a switch between it and my external router. Plugged into that 
> switch is a wireless access point. If someone was on that, the packets 
> would still have to come through the sonicwall to be seen by the 
> iptables system. Sonicwall says the packets could not have come through 
> their device.
> 
> Ok. Next theory: some system on the inside is compromised and spoofing 
> the source address of the packets it sends out. So maybe something on 
> lan2 tried to scan something on lan1. The problem with that is that on 
> the iptables system, I have a rule that says if a packet comes in eth2 
> with a source address other than one from the network attached to that 
> interface, the system is to drop the packet. I have a similar rule for 
> packets coming in eth1 from the other lan. So, that suggests that the 
> packets couldn't have come from the inside, either.
> 
> We have a separate, smaller sonicwall that acts as our vpn concentrator. 
> Only vpn traffic passes through that device. If someone broke in via our 
> vpn, they'd either get a dhcp address on that lan or they could pretend 
> to be something else. If they spoofed packet source from there, again 
> the rule saying drop packets with a source other than that net would 
> kick in.
> 
> So, The ruleset on the sonicwall says it couldn't have come from the 
> outside, and the rulset on the iptables system says it couldn't have 
> come from the inside.
> 
> Now what?
> 
> 	-ste
I have not taken a lot of time to digest your scenario but I'll share a
few spontaneous thoughts.

It is true that RFC 1918 (private) IP addresses do not traverse the
Internet between carriers, however, many carriers use these addresses
internally.  The attack could have come from inside the carrier
network.  Even then, the packets would have to know the path to your
private LAN unless they used source routing or were sitting either on
the network shared by the SonicWall and iptables device or just outside
the SonicWall (since the SonicWall does have a route to your LANs). 
Does your SonicWall let source routed packets through? Were there any
unusual tcp flags set that may have allowed the packet to slip through
the SonicWall? Does the SonicWall advertise its routes? If so, a
wireless intruder would know about them and know that they could be
reached through the SonicWall.

Is it possible the attack did come from a compromised system on the DMZ
and scanned gently enough to not trip the SonicWall port scan detection?
Do you have any HIDS on the DMZ devices to know if they have been
compromised - even something as simple and available as Osiris
(http://osiris.shmoo.com)? Could someone have compromised the SonicWall
itself?

Perhaps the attacker is on the inside - are you sure your anti-spoofing
rules are as tight as you think they are?

You don't say much about where your VPN concentrator drops off its
packets.

Just a couple of random thoughts - I hope they spark something for you -
John
-- 
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net 

Re: incoming interface confusion question

[Jozsef Kadlecsik]

On Mon, 21 Jun 2004, Shaun T. Erickson wrote:

> Shaun T. Erickson wrote:
>
> > Are there any cases where iptables can be confused about what interface
> > a packet came in on? Can a packet arriving on interface A ever be
> > reported as arriving on interface B?
> >
> > I had an incident this weekend, and am trying to be certain that the
> > packets came in the interface my system said it did. It's a Red Hat 9
> > system, running their stock 2.4.20-8 kernel.
>
> Please, can anyone answer this for me? I'm trying to prove or disprove a
> theory that would explain an apparent intrusion incident over the
> weekend. It's very important that I know the definitive answer to this.

It can't happen. I'd investigate further assuming IP address forgery.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: incoming interface confusion question

[Cedric Blancher]

Le lun 21/06/2004 à 19:45, Shaun T. Erickson a écrit :
> Are there any cases where iptables can be confused about what interface 
> a packet came in on? Can a packet arriving on interface A ever be 
> reported as arriving on interface B?

I'm not aware of such a case. Would be quite surprising.

> I had an incident this weekend, and am trying to be certain that the 
> packets came in the interface my system said it did. It's a Red Hat 9 
> system, running their stock 2.4.20-8 kernel.

Do you have any reason to think your system was wrong ?


-- 
http://www.netexit.com/~sid/
PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
>> Hi! I'm your friendly neighbourhood signature virus.
>> Copy me to your signature file and help me spread!

Re: incoming interface confusion question

[Ranjeet Shetye]

* Cedric Blancher (blancher@cartel-securite.fr) wrote:
> Le lun 21/06/2004 à 19:45, Shaun T. Erickson a écrit :
> > Are there any cases where iptables can be confused about what interface 
> > a packet came in on? Can a packet arriving on interface A ever be 
> > reported as arriving on interface B?
> 
> I'm not aware of such a case. Would be quite surprising.
> 
> > I had an incident this weekend, and am trying to be certain that the 
> > packets came in the interface my system said it did. It's a Red Hat 9 
> > system, running their stock 2.4.20-8 kernel.
> 
> Do you have any reason to think your system was wrong ?
> 

In order of likelihood, (highest to lowest):

1) http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.rpf.html

Check your /proc/sys/net/ipv4/conf/$INTF/rp_filter setting.

If it is not set, someone might have tried to attack your network from
the public side, using private src ips. e.g. this can be used for an
attack to make javaVM think that the packets are coming from the host
itself or a trusted server.

2) If it is set, then a VPN might still allow such packets into your
network, past a firewall.

3) If neither, then it might be an issue with the code in Linux.

HTH

-- 
Ranjeet Shetye
Senior Software Engineer
Zultys Technologies
Ranjeet dot Shetye at Zultys dot com
http://www.zultys.com/
 
The views, opinions, and judgements expressed in this message are solely those of
the author. The message contents have not been reviewed or approved by Zultys.