Re: ARP responses broken!

Re: ARP responses broken!

[Eric Weigle]

Hello-

This is a known 'feature' of the Linux kernel, and can help with load sharing
and fault tolerance. However, it can also cause problems (such as when one nic
in a multi-nic machine fails and you don't know right away).

There are three 'solutions' I know of:

  * In recent 2.2 kernels, it was possible to fix this by doing the following as
root: 
        # Start the hiding interface functionality
        echo 1 > /proc/sys/net/ipv4/conf/all/hidden
        # Hide all addresses for this interface
        echo 1 > /proc/sys/net/ipv4/conf/<interface_name>/hidden
    but 2.4 doesn't have that option, for technical reasons.

   * Use 'ifconfig -arp ...' to force an interface not to respond to ARP
requests. Hosts which want to send to that interface may need to manually add
the proper mac address to their ARP tables with 'arp -s'.

   * Use a packet filtering tool (iptables arp filter module, for example) and
just filter the ARP requests and ARP replies so that only the proper set get
through, i.e. when an arp request for the mac address of an interface arrives,
filter out arp replies from all the other interfaces. 

There have been a few threads on this on the linux-kernel mailing list. Search
your favorite archive for them.

-Eric
 
--------------------------------------------
 Eric H. Weigle   CCS-1, RADIANT team
 ehw@lanl.gov     Los Alamos National Lab
 (505) 665-4937   http://home.lanl.gov/ehw/
--------------------------------------------

Broken ARP (was Re: ARP responses broken!)

[Sampsa Ranta]

On Mon, 16 Apr 2001, Eric Weigle wrote:

> Hello-
>
> This is a known 'feature' of the Linux kernel, and can help with load sharing
> and fault tolerance. However, it can also cause problems (such as when one nic
> in a multi-nic machine fails and you don't know right away).

I tought this for a while and this does not help load sharing neighter or
fault tolerance. Causes problem with router environment. I use different
cards to load the problem by assigning different addresses to these and by
pointing these addresses with routes so I use the IP to mark a device.

Only case where this would help with "fault tolerance" is if I
assign address to other device that is not marked as up, it would still
be possible to see the address via other device, and this goes way off.

> There are three 'solutions' I know of:
>
>   * In recent 2.2 kernels, it was possible to fix this by doing the following as
>     but 2.4 doesn't have that option, for technical reasons.

I don't see good valid reasons, I implemented this by myself to the kernel
with  very little effort, only thing is that my patch makes it not
possible to have the same address on multible interfaces, this is
altought very easy to fix. FIB code lacks a way to detect if there are
multible interfaces for a single IP.

>    * Use 'ifconfig -arp ...' to force an interface not to respond to ARP
> requests. Hosts which want to send to that interface may need to manually add
> the proper mac address to their ARP tables with 'arp -s'.

Manual ARP, uh, sounds as nasty as can be.

>    * Use a packet filtering tool (iptables arp filter module, for example) and
> just filter the ARP requests and ARP replies so that only the proper set get
> through, i.e. when an arp request for the mac address of an interface arrives,
> filter out arp replies from all the other interfaces.

There is no ARP module for netfilter and this is such low level issue that
would see requires another way to be solved by all means.

It is not working correctly if there are two ARP replies for a ARP query,
another one from interface that DOES NOT have the address assigned to it.
Basically I should have a firewall rule that would instantly DROP all
packets that come to the interface with address that it does not have if I
don't have forwarding enabled.

> There have been a few threads on this on the linux-kernel mailing list. Search
> your favorite archive for them.

Well, this is not solved so we probaply need one more.

The code I used to do the trick at my network was as simple as this,
in function arp_rcv, the problem is ip_dev_find that does know if there
are other devices with same IP address.

--- job/arp.c   Tue Apr 17 17:05:48 2001
+++ arp.c       Tue Apr 17 17:05:24 2001
@@ -571,6 +571,7 @@
        int addr_type;
        struct in_device *in_dev = in_dev_get(dev);
        struct neighbour *n;
+       struct net_device *devx = NULL;

 /*
  *     The hardware length of the packet should match the hardware length
@@ -728,8 +729,10 @@

                rt = (struct rtable*)skb->dst;
                addr_type = rt->rt_type;

-               if (addr_type == RTN_LOCAL) {
+               devx = ip_dev_find(tip);
+               if (addr_type == RTN_LOCAL &&
+                   devx == dev) {
                        n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
                        if (n) {
                                arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);


 - Sampsa Ranta
   sampsa@netsonic.fi

Re: ARP responses broken!

[Andi Kleen]

On Mon, Apr 16, 2001 at 03:26:19PM -0600, Eric Weigle wrote:
> Hello-
> 
> This is a known 'feature' of the Linux kernel, and can help with load sharing
> and fault tolerance. However, it can also cause problems (such as when one nic
> in a multi-nic machine fails and you don't know right away).
> 
> There are three 'solutions' I know of:
> 
>   * In recent 2.2 kernels, it was possible to fix this by doing the following as

Or use arpfilter in even newer 2.2 kernels; which filters based on the routing
table. "hidden" is quite a sledgehammer which often does more harm than good.


-Andi

Re: ARP responses broken!

[Martin Josefsson]

On Tue, 17 Apr 2001, Andi Kleen wrote:

> On Mon, Apr 16, 2001 at 03:26:19PM -0600, Eric Weigle wrote:
> > Hello-
> > 
> > This is a known 'feature' of the Linux kernel, and can help with load sharing
> > and fault tolerance. However, it can also cause problems (such as when one nic
> > in a multi-nic machine fails and you don't know right away).
> > 
> > There are three 'solutions' I know of:
> > 
> >   * In recent 2.2 kernels, it was possible to fix this by doing the following as
> 
> Or use arpfilter in even newer 2.2 kernels; which filters based on the routing
> table. "hidden" is quite a sledgehammer which often does more harm than good.

Does arpfilter exist in 2.4 kernels?

/Martin

Re: ARP responses broken!

[Andi Kleen]

On Tue, Apr 17, 2001 at 04:53:01PM +0200, Martin Josefsson wrote:
> On Tue, 17 Apr 2001, Andi Kleen wrote:
> 
> > On Mon, Apr 16, 2001 at 03:26:19PM -0600, Eric Weigle wrote:
> > > Hello-
> > > 
> > > This is a known 'feature' of the Linux kernel, and can help with load sharing
> > > and fault tolerance. However, it can also cause problems (such as when one nic
> > > in a multi-nic machine fails and you don't know right away).
> > > 
> > > There are three 'solutions' I know of:
> > > 
> > >   * In recent 2.2 kernels, it was possible to fix this by doing the following as
> > 
> > Or use arpfilter in even newer 2.2 kernels; which filters based on the routing
> > table. "hidden" is quite a sledgehammer which often does more harm than good.
> 
> Does arpfilter exist in 2.4 kernels?

Not yet, will be merged very soon. I can send you a patch if you need it urgently.


-Andi

Re: ARP responses broken!

[Martin Josefsson]

On Tue, 17 Apr 2001, Andi Kleen wrote:

[snip]
> > Does arpfilter exist in 2.4 kernels?
> 
> Not yet, will be merged very soon. I can send you a patch if you need it urgently.

No I don't need it urgently.
I was asking because I had this problem before (router with two cards
against one physical subnet) and arpwatch complained that the router kept
switching MACaddresses all the time.

/Martin

Re: Broken ARP (was Re: ARP responses broken!)

[Eric Weigle]

Ok, I was ignorant of the arp filter functionality in 2.2. I found an old
(probably painfully out-of-date) posting the patch Andi Kleen was referring to
in the archive, but I've not used it.
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.2/1198.html

> I tought this for a while and this does not help load sharing neighter or
> fault tolerance. Causes problem with router environment. I use different
> cards to load the problem by assigning different addresses to these and by
> pointing these addresses with routes so I use the IP to mark a device.
> 
> Only case where this would help with "fault tolerance" is if I
> assign address to other device that is not marked as up, it would still
> be possible to see the address via other device, and this goes way off.

Indeed, the default behavior does cause problems in a router environment, but
this only happens when multiple nics are on the same subnet; in a 'true' router
each nic would be on a separate subnet. Regardless, I personally use FreeBSD
when I need a router (Horrors! ;)

What I meant by load sharing was implicit sharing rather than explicit sharing;
when an ARP request comes the reply the host gets may have the MAC address of
NICs other than the one explicitly bound to the given IP-- thus different hosts
will semi-randomly get different MAC addresses and thus send to different NICs;
this implicit sharing completely hoses explicit load sharing.

And as for fault tolerance, here's what happened to me, as I mentioned in
another message: We have a 8-node cluster with 2 nics, a eepro and a gig-e
acenic in each node. A while back the acenic driver had some problems and would
silently fail after a while; the arp reponse behavior allowed the cluster to
remain 'up' long enough to finish the jobs we assigned to it (although
performance sucked since all traffic went over the eepros). After we were done,
we could ifdown/ifup the interfaces and all was good. Again, this is a sort of
'implicit' fault tolerance, rather than a more explicit form where the card goes
down, we get some sort of notification, and it fails over to the other card
explicitly.

> The code I used to do the trick at my network was as simple as this,
> in function arp_rcv, the problem is ip_dev_find that does know if there
> are other devices with same IP address.
Well, Yes, but that's not really the issue. The problem is 'what is the proper
*default* behavior of the Linux ARP subsystem'... this code changes it, which is
probably more of a political than technical decision. Where you (and I) see
'broken' others see 'feature' :/

-Eric

--------------------------------------------
 Eric H. Weigle   CCS-1, RADIANT team
 ehw@lanl.gov     Los Alamos National Lab
 (505) 665-4937   http://home.lanl.gov/ehw/
--------------------------------------------

Re: Broken ARP (was Re: ARP responses broken!)

[dean gaudet]

On Tue, 17 Apr 2001, Eric Weigle wrote:

> Ok, I was ignorant of the arp filter functionality in 2.2. I found an old
> (probably painfully out-of-date) posting the patch Andi Kleen was referring to
> in the archive, but I've not used it.
> http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.2/1198.html

oh heck, why not post a third patch for this problem.  here's the one i
wrote.

-dean

--- linux/net/ipv4/arp.c.badproxy	Mon Feb 12 17:28:48 2001
+++ linux/net/ipv4/arp.c	Tue Feb 13 20:06:37 2001
@@ -737,10 +737,12 @@
 		addr_type = rt->rt_type;

 		if (addr_type == RTN_LOCAL) {
+			if ((rt->rt_flags&RTCF_DIRECTSRC) || IN_DEV_PROXY_ARP(in_dev)) {
 			n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
 			if (n) {
 				arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);
 				neigh_release(n);
+			}
 			}
 			goto out;
 		} else if (IN_DEV_FORWARD(in_dev)) {

Re: ARP responses broken!

[Alan Cox]

> I was asking because I had this problem before (router with two cards
> against one physical subnet) and arpwatch complained that the router kept
> switching MACaddresses all the time.

That sounds like a bug in arpwatch. A box can have multiple mac addresses. Its
probably a tricky one to handle but arpwatch I guess should spot and cope with
repeated transitions between the same set of addresses as one warning

Re: ARP responses broken!

[Rogier Wolff]

Alan Cox wrote:
> > I was asking because I had this problem before (router with two cards
> > against one physical subnet) and arpwatch complained that the router kept
> > switching MACaddresses all the time.
 
> That sounds like a bug in arpwatch. A box can have multiple mac
> addresses. Its probably a tricky one to handle but arpwatch I guess
> should spot and cope with repeated transitions between the same set
> of addresses as one warning

Well, two. Or three. 

- Hey, IP x changed from mac X to mac Y. 
- Hey, IP x changed back again to X. 
- Hmm. IP X seems to be using both Mac X and and Mac Y. 
	No further warnings will be issued about this. 

If someone is taking over an IP address (which is especially what
arpwatch should be looking for), this is exactly what you'll see. Having
the issue be ignored after one warning is bad. 

Oh, and I know people who swear that this would be an invalid
configuration, so that it is good for arpwatch to should loud and
clear about it...

				Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots. 

Re: Broken ARP (was Re: ARP responses broken!)

[Julian Anastasov]

	Hello,

Sampsa Ranta wrote:

> The code I used to do the trick at my network was as simple as this,
> in function arp_rcv, the problem is ip_dev_find that does know if there
> are other devices with same IP address.

	I don't think this is your problem. You patch is not correct.
In fact, you implement the same function as in "hidden" but you are
missing some things. Please, read the "hidden" flag description in
the kernel docs. You must solve the case where your ARP probes are sent
always through one device due to your routing (this is out traffic,
yes?). These probes soon or later will cause you problems because
they change the entry in the remote hosts' ARP tables. You so carefully
tried to advertise the address on specific interface and now the other
hosts again talk to one card only.

	who-has 194.29.192.1 tell 194.29.192.38

	and your are dead :)

	So, please try "hidden" before going into more problems with
these patches (I see two in this thread, and I saw so many before).


Regards

--
Julian Anastasov <ja@ssi.bg>

Re: Broken ARP (was Re: ARP responses broken!)

[Sampsa Ranta]

On Wed, 18 Apr 2001, Julian Anastasov wrote:

>
> 	Hello,
>
> Sampsa Ranta wrote:
>
> > The code I used to do the trick at my network was as simple as this,
> > in function arp_rcv, the problem is ip_dev_find that does know if there
> > are other devices with same IP address.
>
> 	I don't think this is your problem. You patch is not correct.
> In fact, you implement the same function as in "hidden" but you are
> missing some things. Please, read the "hidden" flag description in
> the kernel docs. You must solve the case where your ARP probes are sent
> always through one device due to your routing (this is out traffic,
> yes?). These probes soon or later will cause you problems because
> they change the entry in the remote hosts' ARP tables. You so carefully
> tried to advertise the address on specific interface and now the other
> hosts again talk to one card only.
>
> 	who-has 194.29.192.1 tell 194.29.192.38
>
> 	and your are dead :)
>
> 	So, please try "hidden" before going into more problems with
> these patches (I see two in this thread, and I saw so many before).

The problem is there is no hidden in 2.4 and my patch did not modify
behavior of sending ARP requests and receiving the replies, are there
issues to be solved in these cases, too?

 - Sampsa Ranta
   sampsa@netsonic.fi

Re: Broken ARP (was Re: ARP responses broken!)

[Sampsa Ranta]

On Wed, 18 Apr 2001, Julian Anastasov wrote:

>
> 	Hello,
>
> Sampsa Ranta wrote:
>
> > The code I used to do the trick at my network was as simple as this,
> > in function arp_rcv, the problem is ip_dev_find that does know if there
> > are other devices with same IP address.
>
> 	I don't think this is your problem. You patch is not correct.
> In fact, you implement the same function as in "hidden" but you are
> missing some things. Please, read the "hidden" flag description in
> the kernel docs. You must solve the case where your ARP probes are sent
> always through one device due to your routing (this is out traffic,
> yes?). These probes soon or later will cause you problems because
> they change the entry in the remote hosts' ARP tables. You so carefully
> tried to advertise the address on specific interface and now the other
> hosts again talk to one card only.
>
> 	who-has 194.29.192.1 tell 194.29.192.38
>
> 	and your are dead :)
>
> 	So, please try "hidden" before going into more problems with
> these patches (I see two in this thread, and I saw so many before).

Are you referring to the arp_solicit()?

       if (skb && inet_addr_type(skb->nh.iph->saddr) == RTN_LOCAL)
                saddr = skb->nh.iph->saddr;
       else
                saddr = inet_select_addr(dev, target, RT_SCOPE_LINK);

Why is this hidden flag removed in 2.4 series with replacing no
functionality that would help to solve the problems?

 - Sampsa Ranta
   sampsa@netsonic.fi

Re: Broken ARP (was Re: ARP responses broken!)

[Sampsa Ranta]

I have a rathar strange way to first ask and then try to find the answer
on my own. But what I found was:

from http://www.appwatch.com/lists/linux-kernel/Week-of-Mon-20010122/018588.html
> > am most curious about is how it ending up being removed from the kernel
> > in the first place.  It must have been a decision that someone made.
> > Either, we don't need that any more since we can do it this way, or
> > we'll take it out since nobody uses it.
>
> It was only submitted to 2.2 a few months ago (=years after 2.3 branched), but
> never added to 2.4.

So I wonder if this hidden feature or alike should be brought to 2.4 tree
also?

 - Sampsa Ranta
   sampsa@netsonic.fi

Re: Broken ARP (was Re: ARP responses broken!)

[Julian Anastasov]

	Hello,

On Thu, 19 Apr 2001, Sampsa Ranta wrote:

> So I wonder if this hidden feature or alike should be brought to 2.4 tree
> also?

	The three flags that can control the ARP behavior in 2.2
(arp_filter, hidden and rp_filter) cover almost everything without
breaking any RFC826 rule. You can always find the missing from Linux 2.4
functionality in the LVS site, where it is really used.

	I hope the following posts/threads explain almost everything
related :)

http://marc.theaimsgroup.com/?l=linux-kernel&m=98032243112274&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=98042063530177&w=2
http://marc.theaimsgroup.com/?t=98019795800013&w=2&r=1
http://marc.theaimsgroup.com/?t=95743539800002&w=2&r=1


Regards

--
Julian Anastasov <ja@ssi.bg>