[LARTC] Proxy ARP with a Coyote Point equalizer

[LARTC] Proxy ARP with a Coyote Point equalizer

[Greg Scott]

Here is a puzzle.  
 
I have a network with several servers. It's a mess.  It's a /24 and
pieces and servers are all over the place inside this /24 block, on both
sides of the firewall.  For example, the router at 1.2.3.1 is outside
the firewall and many of the servers at 1.2.3.nnn/24 are behind the
firewall.  (Obviously, 1.2.3.nnn is a fudged network.)
 
eth0 points outward to the Internet. 
eth1 points inward to the serers.  
 
Both eth0 and eth1 have IP Address 1.2.3.2.  I setup  proxy ARP like
this:
 
echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp

And I set up appropriate routes to the systems on both sides of the
firewall.  

This all works - all the systems route the way they are supposed to
route.  

Here is the problem.  Behind the firewall is a Coyote Point Equalizer at
1.2.3.10, with a high-volume website behind it spread across several
servers.  Every time I put this proxy ARP firewall in place, that nasty
Coyote Point box dies and this breaks the high volume website behind it
and makes lots of people mad.  I've never seen a Coyote Point Equalizer
but I have a hunch it might not get along well with a proxy ARP device
in its same network.  

Here are my questions:

Proxy ARP really means proxy ARP - that firewall answers ARP requests
for anything and everything it sees, for any network.  This also has
consequences for new devices that try to be polite when they set IP
Addresses for themselves by ARPing to see if anyone else answers at that
address.  Is there a way to limit proxy ARP to a list of IP Addresses?  

Or - should I forget proxy ARP and look at bridging instead?  Can I do
bridging and still access the bridged interfaces remotely?  

Thanks

- Greg Scott
  GregScott@InfraSupportEtc.com

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Proxy ARP with a Coyote Point equalizer

[Grant Taylor]

On 5/30/2007 6:46 PM, Greg Scott wrote:
> Here is the problem.  Behind the firewall is a Coyote Point Equalizer 
> at 1.2.3.10, with a high-volume website behind it spread across 
> several servers.  Every time I put this proxy ARP firewall in place, 
> that nasty Coyote Point box dies and this breaks the high volume 
> website behind it and makes lots of people mad.  I've never seen a 
> Coyote Point Equalizer but I have a hunch it might not get along well 
> with a proxy ARP device in its same network.

Hrm...

> Proxy ARP really means proxy ARP - that firewall answers ARP requests 
> for anything and everything it sees, for any network.  This also has 
> consequences for new devices that try to be polite when they set IP 
> Addresses for themselves by ARPing to see if anyone else answers at 
> that address.  Is there a way to limit proxy ARP to a list of IP 
> Addresses?

This will be Proxy ARP implementation specific.  I have no idea whether 
or not Linux can be configured to behave as you are asking or not.

> Or - should I forget proxy ARP and look at bridging instead?

Having just (briefly) brushed up on Proxy ARPing, I can see how it would 
be a problem for a load balancer.  Most load balancers work on a couple 
of different levels, either IP <-> MAC spoofing, or NATing.  The former 
method is probably what is happening and thus having a problem with your 
Proxy ARP router / firewall.

Consider if you will a host out side of the Proxy ARP router / firewall 
trying to connect to an IP address that is both behind the Proxy ARP 
router / firewall AND the load balancer.  If the load balancer changes 
the MAC address that the IP address belongs to, the Proxy ARP router / 
firewall will inevitably end up pointing to the wrong internal MAC.  How 
will the load balancer handle the traffic when it does not start flowing 
to the alternative MAC like it wants?  I can not say.  But, I do see a 
very big potential for a conflict.  In said conflict, I can not say any 
thing to how any of the equipment will fail.  Thus, you could end up in 
the scenario you are in now.

I can't say for sure as to whether or not you should forget about proxy 
ARP or not, but I can say for sure that bridging will do what you are 
wanting to do very well.

Bridging will pass the ARP requests in directly to the load balancer 
like it is expecting so that it can control things the way that it wants 
to.  This means that when the load balancer alters the IP <-> MAC 
mapping, the upstream device on the other side of the bridge will see 
the changed MAC address.

I think I would go the bridging route.

> Can I do bridging and still access the bridged interfaces remotely?

Most definitely!  Put your IP address on the bridge interface.

I.e. eth0 and eth1 are bridged together by br0.

ifconfig br0 1.2.3.2 netmask 255.255.255.0

You will be able to access 1.2.3.2 from either side of the bridge.  That 
is presuming that you do not use EBTables / IPTables to filter the 
bridged traffic.  In other words, so long as you are not doing any layer 
2 filtering yes.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Proxy ARP with a Coyote Point equalizer

[gypsy]

Greg Scott wrote:
> 
> Here is a puzzle.
> 
> I have a network with several servers. It's a mess.  It's a /24 and
> pieces and servers are all over the place inside this /24 block, on both
> sides of the firewall.  For example, the router at 1.2.3.1 is outside
> the firewall and many of the servers at 1.2.3.nnn/24 are behind the
> firewall.  (Obviously, 1.2.3.nnn is a fudged network.)
> 
> eth0 points outward to the Internet.
> eth1 points inward to the serers.
> 
> Both eth0 and eth1 have IP Address 1.2.3.2.  I setup  proxy ARP like
> this:
> 
> echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
> echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp
> 
> And I set up appropriate routes to the systems on both sides of the
> firewall.
> 
> This all works - all the systems route the way they are supposed to
> route.
> 
> Here is the problem.  Behind the firewall is a Coyote Point Equalizer at
> 1.2.3.10, with a high-volume website behind it spread across several
> servers.  Every time I put this proxy ARP firewall in place, that nasty
> Coyote Point box dies and this breaks the high volume website behind it
> and makes lots of people mad.  I've never seen a Coyote Point Equalizer
> but I have a hunch it might not get along well with a proxy ARP device
> in its same network.
> 
> Here are my questions:
> 
> Proxy ARP really means proxy ARP - that firewall answers ARP requests
> for anything and everything it sees, for any network.  This also has
> consequences for new devices that try to be polite when they set IP
> Addresses for themselves by ARPing to see if anyone else answers at that
> address.  Is there a way to limit proxy ARP to a list of IP Addresses?
> 
> Or - should I forget proxy ARP and look at bridging instead?  Can I do
> bridging and still access the bridged interfaces remotely?
> 
> Thanks
> 
> - Greg Scott
>   GregScott@InfraSupportEtc.com

See http://yesican.chsoft.biz/lartc/proxy-arp.conf
and http://yesican.chsoft.biz/lartc/proxy-arp.sh
to see if that helps.  The LAN interface (eth0) uses the
/proc-/proxy_arp setting while the WAN (eth1) interface uses the script.

FWIW, those are my working setups.  One computer has a WAN connection
(eth1) and all other servers inside connect to its eth0.  The above
script and config file are on that computer.  Note that both eth1 and
eth0 have the same IP (66.209.101.198) and netmask.  This machine has a
third interface (eth2) to the LAN, but that is not material here.

If the ISP changes things, which they have done a couple of times, I
have to ask them to flush their ARP cache manually because their
retention is HUGE (~70 minutes), but except for that, I've never had any
problems with this setup.  I had no success at all trying to use /proc
on eth1.
--
gypsy
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Proxy ARP with a Coyote Point equalizer

[Greg Scott]

I was thinking about what Grant said in his earlier reply about
bridging, and the more I think about it, the more I think Grant is
right.  I was watching tcpdump and some firewall accounting rules when
we tried this the other day and I saw all kinds of traffic that had
absolutely nothing to do with my network - and I was blocking it!  And I
kept wondering, why was I blocking traffic that had a destination IP
address nowhere near me that I should not care about?  

Well, this is a co-location site, so an Ethernet connects this network
to the Internet, not a point to point serial or any kind of WAN.  No
doubt lots of other folks had their networks in racks in the same room
on the same extended Ethernet as my stuff.  

But still, why did my box care about any of this traffic at all?  Why
would I specifically block it - why didn't the NIC in my box just ignore
it, the way most normal systems do in an Ethernet for traffic it doesn't
care about?  Well, duh!  It's proxy ARP.  Every time anyone, anywhere on
this Ethernet, sends out an ARP request and I see it, I answer -  yup,
here is my MAC Address and it belongs to the IP Address you just asked
about.  I don't care what IP Address, I answer ARP requests for ALL IP
Addresses!  I was essentialy ARP spoofing the whole world!  Well, at
least the whole world on that Ethernet that morning.

Holy moley - based on that analysis, proxy ARP should be outlawed from
any co-location site, at least anything directly exposed to all the
public traffic.  For that few minutes, I'll bet I messed up systems
belonging to who-knows-how-many customers!  Fortunately, it was in the
middle of the night in my corner of the world.

For Gypsy - it's not my own ARP cache I was messing with, it was
everyone else's ARP caches.  

Anyway, lesson learned.  Maybe this writeup will help somebody else out
there.  Definitely do bridging.

- Greg Scott



-----Original Message-----
From: gypsy [mailto:gypsy@iswest.com] 
Sent: Thursday, May 31, 2007 1:50 AM
To: lartc@mailman.ds9a.nl
Cc: Greg Scott
Subject: Re: [LARTC] Proxy ARP with a Coyote Point equalizer

Greg Scott wrote:
> 
> Here is a puzzle.
> 
> I have a network with several servers. It's a mess.  It's a /24 and 
> pieces and servers are all over the place inside this /24 block, on 
> both sides of the firewall.  For example, the router at 1.2.3.1 is 
> outside the firewall and many of the servers at 1.2.3.nnn/24 are 
> behind the firewall.  (Obviously, 1.2.3.nnn is a fudged network.)
> 
> eth0 points outward to the Internet.
> eth1 points inward to the serers.
> 
> Both eth0 and eth1 have IP Address 1.2.3.2.  I setup  proxy ARP like
> this:
> 
> echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
> echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp
> 
> And I set up appropriate routes to the systems on both sides of the 
> firewall.
> 
> This all works - all the systems route the way they are supposed to 
> route.
> 
> Here is the problem.  Behind the firewall is a Coyote Point Equalizer 
> at 1.2.3.10, with a high-volume website behind it spread across 
> several servers.  Every time I put this proxy ARP firewall in place, 
> that nasty Coyote Point box dies and this breaks the high volume 
> website behind it and makes lots of people mad.  I've never seen a 
> Coyote Point Equalizer but I have a hunch it might not get along well 
> with a proxy ARP device in its same network.
> 
> Here are my questions:
> 
> Proxy ARP really means proxy ARP - that firewall answers ARP requests 
> for anything and everything it sees, for any network.  This also has 
> consequences for new devices that try to be polite when they set IP 
> Addresses for themselves by ARPing to see if anyone else answers at 
> that address.  Is there a way to limit proxy ARP to a list of IP
Addresses?
> 
> Or - should I forget proxy ARP and look at bridging instead?  Can I do

> bridging and still access the bridged interfaces remotely?
> 
> Thanks
> 
> - Greg Scott
>   GregScott@InfraSupportEtc.com

See http://yesican.chsoft.biz/lartc/proxy-arp.conf
and http://yesican.chsoft.biz/lartc/proxy-arp.sh
to see if that helps.  The LAN interface (eth0) uses the
/proc-/proxy_arp setting while the WAN (eth1) interface uses the script.

FWIW, those are my working setups.  One computer has a WAN connection
(eth1) and all other servers inside connect to its eth0.  The above
script and config file are on that computer.  Note that both eth1 and
eth0 have the same IP (66.209.101.198) and netmask.  This machine has a
third interface (eth2) to the LAN, but that is not material here.

If the ISP changes things, which they have done a couple of times, I
have to ask them to flush their ARP cache manually because their
retention is HUGE (~70 minutes), but except for that, I've never had any
problems with this setup.  I had no success at all trying to use /proc
on eth1.
--
gypsy
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Proxy ARP with a Coyote Point equalizer

[Grant Taylor]

On 05/31/07 13:41, Greg Scott wrote:
> I was thinking about what Grant said in his earlier reply about 
> bridging, and the more I think about it, the more I think Grant is 
> right.  I was watching tcpdump and some firewall accounting rules 
> when we tried this the other day and I saw all kinds of traffic that 
> had absolutely nothing to do with my network - and I was blocking it! 
> And I kept wondering, why was I blocking traffic that had a 
> destination IP address nowhere near me that I should not care about?

That's not good.

> Well, this is a co-location site, so an Ethernet connects this 
> network to the Internet, not a point to point serial or any kind of 
> WAN.  No doubt lots of other folks had their networks in racks in the 
> same room on the same extended Ethernet as my stuff.

Probably.  However I have to ask, why are so many different networks 
sharing one broadcast domain?  Or could it be that traffic was trying to 
be routed to the target subnet, but your system responded to the ARP for 
the IP of the target router???  I wonder...  Either way, this is not good.

This is also why I have seriously considered statically setting some IP 
to MAC address mappings in some more stringent environments.  If the 
upstream router knows the MAC address of my router, it will not need to 
ARP for it and thus I do not care if someone else claims to use my IP or 
not because the router will know which MAC address to use.

> But still, why did my box care about any of this traffic at all?  Why 
> would I specifically block it - why didn't the NIC in my box just 
> ignore it, the way most normal systems do in an Ethernet for traffic 
> it doesn't care about?  Well, duh!  It's proxy ARP.  Every time 
> anyone, anywhere on this Ethernet, sends out an ARP request and I see 
> it, I answer -  yup, here is my MAC Address and it belongs to the IP 
> Address you just asked about.  I don't care what IP Address, I answer 
> ARP requests for ALL IP Addresses!  I was essentially ARP spoofing the 
> whole world!  Well, at least the whole world on that Ethernet that 
> morning.

Oh NO!  That was not a good thing.  I wonder how many people you 
effected while doing that.

> Holy moley - based on that analysis, proxy ARP should be outlawed 
> from any co-location site, at least anything directly exposed to all 
> the public traffic.  For that few minutes, I'll bet I messed up 
> systems belonging to who-knows-how-many customers!  Fortunately, it 
> was in the middle of the night in my corner of the world.

Just because it is the middle of the night for you does not mean that it 
was not 9:30 in the morning for someone else who potentially has their 
box co-located there.  I think I would be tempted to contact your 
co-location provider and let them know what happened.  Consider if you 
were there administrator trying to figure out why a bunch of clients 
traffic did not route correctly for a short time and then repaired its 
self with out any explanation at all.  I know that I would want to know 
if such a thing happened in one of my data centers.  At the very least I 
would want to know that it was not a bug in the router but rather 
something that is a valid explanation for what happened.

As far as outlawing Proxy ARP, I don't know.  I do know that I see no 
reason to use Proxy ARP when bridging is sow much more powerful and 
safer.  Just think, you an do any and all IPTables (layer 3 and above) 
firewalling ON a layer 2 device.  You can be REALLY devious with this. 
If course there is also EBTables and ARPTables too.

> For Gypsy - it's not my own ARP cache I was messing with, it was 
> everyone else's ARP caches.

Gulp!

> Anyway, lesson learned.  Maybe this writeup will help somebody else 
> out there.  Definitely do bridging.

Here! HERE!



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc