ARP responses broken!

ARP responses broken!

[Sampsa Ranta]

Hello, I had a mystery with my Linux running 2.4.2 kernel with ARP packet
response.

I have two interfaces that share same subnet, I call eth0 194.29.192.37
and eth1 194.29.192.38. I have forwarding turned on, proxy arp is not
neighter are redirects.

When I flush local neighbor table in other machine I use to observe the
response and ping the router I get response like:

23:38:25.278848 > arp who-has 194.29.192.38 tell 194.29.192.10 (0:50:da:82:ae:9f)
23:38:25.278988 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:64 (0:50:da:82:ae:9f)
23:38:25.279009 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:6c (0:50:da:82:ae:9f)

The second one is the valid one, but both interfaces seem to answer to the
broadcasted packet with their own ARP addresses.

This came up when I wondered why I get responses to wrong interface with
Zebra.

 - Sampsa Ranta
   sampsa@netsonic.fi

Re: ARP responses broken!

[Eric Weigle]

Hello-

This is a known 'feature' of the Linux kernel, and can help with load sharing
and fault tolerance. However, it can also cause problems (such as when one nic
in a multi-nic machine fails and you don't know right away).

There are three 'solutions' I know of:

  * In recent 2.2 kernels, it was possible to fix this by doing the following as
root: 
        # Start the hiding interface functionality
        echo 1 > /proc/sys/net/ipv4/conf/all/hidden
        # Hide all addresses for this interface
        echo 1 > /proc/sys/net/ipv4/conf/<interface_name>/hidden
    but 2.4 doesn't have that option, for technical reasons.

   * Use 'ifconfig -arp ...' to force an interface not to respond to ARP
requests. Hosts which want to send to that interface may need to manually add
the proper mac address to their ARP tables with 'arp -s'.

   * Use a packet filtering tool (iptables arp filter module, for example) and
just filter the ARP requests and ARP replies so that only the proper set get
through, i.e. when an arp request for the mac address of an interface arrives,
filter out arp replies from all the other interfaces. 

There have been a few threads on this on the linux-kernel mailing list. Search
your favorite archive for them.

-Eric
 
--------------------------------------------
 Eric H. Weigle   CCS-1, RADIANT team
 ehw@lanl.gov     Los Alamos National Lab
 (505) 665-4937   http://home.lanl.gov/ehw/
--------------------------------------------

Re: ARP responses broken!

[Bernd Eckenfels]

In article <Pine.LNX.4.33.0104162335170.30406-100000@nalle.netsonic.fi> you wrote:
> The second one is the valid one, but both interfaces seem to answer to the
> broadcasted packet with their own ARP addresses.

it is because the kernel does not know if both interfaces are on one subnet,
or not. The easisets thing to solve this is t use the MAC moduleof netfilter
and deny the incoming requests/responsnes based on the ip.

Greetings
Bernd

Re: ARP responses broken!

[Eric Weigle]

> oh great, now I wont be able to upgrade our kernels to 2.4 unless I find a
> utility to filter out the ARP requests?
"There's more than one way to do it" (see below)

> Why was this ability removed?
Apparently the decision was made to do it this way because it simplified the
fast path of the code; but I could be wrong. I have yet to achieve Linux Guru
status, right now I'm just some guy who has hit this problem and knows the
work-arounds.

> If I screw up and put it on the wrong card, I WANT the system to stop
> working...
I agree. I'm not making an argument for this implementation. Well, maybe I am;
Linux is intended as a desktop operating system and in that context most people
would rather have the failsafe than the failstop. Perhaps this method is
`inelegant' but it kept up a cluster of machines we have here a lot longer than
they might otherwise have functioned (the gige Acenic driver had some problems a
while back and about half the cards silently failed-- the 'broken' arp responses
meant that we could still talk to those boxes without reconfiguring IPs).

Anyway, Here's links to a discussion that occurred this January.

If you require the 'hidden' functionality, the first message cites the following
links; the patch apparently works with 2.4.x with some tweaking.
    http://www.linuxvirtualserver.org/arp.html
    http://www.linuxvirtualserver.org/hidden-2.3.41-1.diff

http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0014.html
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0020.html
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0188.html
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0213.html
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0220.html
    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0268.html
        http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0334.html
            http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0410.html

Hope that helps
-Eric

--------------------------------------------
 Eric H. Weigle   CCS-1, RADIANT team
 ehw@lanl.gov     Los Alamos National Lab
 (505) 665-4937   http://home.lanl.gov/ehw/
--------------------------------------------

Re: ARP responses broken!

[Andi Kleen]

On Mon, Apr 16, 2001 at 03:26:19PM -0600, Eric Weigle wrote:
> Hello-
> 
> This is a known 'feature' of the Linux kernel, and can help with load sharing
> and fault tolerance. However, it can also cause problems (such as when one nic
> in a multi-nic machine fails and you don't know right away).
> 
> There are three 'solutions' I know of:
> 
>   * In recent 2.2 kernels, it was possible to fix this by doing the following as

Or use arpfilter in even newer 2.2 kernels; which filters based on the routing
table. "hidden" is quite a sledgehammer which often does more harm than good.


-Andi

Re: ARP responses broken!

[Christopher Friesen]

Sampsa Ranta wrote:

> I have two interfaces that share same subnet, I call eth0 194.29.192.37
> and eth1 194.29.192.38. I have forwarding turned on, proxy arp is not
> neighter are redirects.
> 
> When I flush local neighbor table in other machine I use to observe the
> response and ping the router I get response like:
> 
> 23:38:25.278848 > arp who-has 194.29.192.38 tell 194.29.192.10 (0:50:da:82:ae:9f)
> 23:38:25.278988 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:64 (0:50:da:82:ae:9f)
> 23:38:25.279009 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:6c (0:50:da:82:ae:9f)
> 
> The second one is the valid one, but both interfaces seem to answer to the
> broadcasted packet with their own ARP addresses.

This is the default Linux behaviour.  It can be turned off by running the
following command as root:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

This ensures that interfaces will only respond to arp requests for IP addresses
which are configured as belonging to that particular interface.

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: ARP responses broken!

[Bernd Eckenfels]

> but why would you want it to reply for the IP of the other interface even if
> it was NOT on the same subnet?

Because Linux is always answering to all its local IP addresses, regardless
of the Network interface. Even if you tun off the IP Forwarding.

This is by Designs, there are situation where this is good and there are
situations where it is not so good. But it's an old tradition.

Greetings
Bernd
-- 
  (OO)      -- Bernd_Eckenfels@Wendelinusstrasse39.76646Bruchsal.de --
 ( .. )  ecki@{inka.de,linux.de,debian.org} http://home.pages.de/~eckes/
  o--o     *plush*  2048/93600EFD  eckes@irc  +497257930613  BE5-RIPE
(O____O)  When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!

Re: ARP responses broken!

[Martin Josefsson]

On Tue, 17 Apr 2001, Andi Kleen wrote:

> On Mon, Apr 16, 2001 at 03:26:19PM -0600, Eric Weigle wrote:
> > Hello-
> > 
> > This is a known 'feature' of the Linux kernel, and can help with load sharing
> > and fault tolerance. However, it can also cause problems (such as when one nic
> > in a multi-nic machine fails and you don't know right away).
> > 
> > There are three 'solutions' I know of:
> > 
> >   * In recent 2.2 kernels, it was possible to fix this by doing the following as
> 
> Or use arpfilter in even newer 2.2 kernels; which filters based on the routing
> table. "hidden" is quite a sledgehammer which often does more harm than good.

Does arpfilter exist in 2.4 kernels?

/Martin

Re: ARP responses broken!

[Andi Kleen]

On Tue, Apr 17, 2001 at 04:53:01PM +0200, Martin Josefsson wrote:
> On Tue, 17 Apr 2001, Andi Kleen wrote:
> 
> > On Mon, Apr 16, 2001 at 03:26:19PM -0600, Eric Weigle wrote:
> > > Hello-
> > > 
> > > This is a known 'feature' of the Linux kernel, and can help with load sharing
> > > and fault tolerance. However, it can also cause problems (such as when one nic
> > > in a multi-nic machine fails and you don't know right away).
> > > 
> > > There are three 'solutions' I know of:
> > > 
> > >   * In recent 2.2 kernels, it was possible to fix this by doing the following as
> > 
> > Or use arpfilter in even newer 2.2 kernels; which filters based on the routing
> > table. "hidden" is quite a sledgehammer which often does more harm than good.
> 
> Does arpfilter exist in 2.4 kernels?

Not yet, will be merged very soon. I can send you a patch if you need it urgently.


-Andi

Re: ARP responses broken!

[Martin Josefsson]

On Tue, 17 Apr 2001, Andi Kleen wrote:

[snip]
> > Does arpfilter exist in 2.4 kernels?
> 
> Not yet, will be merged very soon. I can send you a patch if you need it urgently.

No I don't need it urgently.
I was asking because I had this problem before (router with two cards
against one physical subnet) and arpwatch complained that the router kept
switching MACaddresses all the time.

/Martin

Re: ARP responses broken!

[Alan Cox]

> I was asking because I had this problem before (router with two cards
> against one physical subnet) and arpwatch complained that the router kept
> switching MACaddresses all the time.

That sounds like a bug in arpwatch. A box can have multiple mac addresses. Its
probably a tricky one to handle but arpwatch I guess should spot and cope with
repeated transitions between the same set of addresses as one warning

RE: ARP responses broken!

[Bingner Sam J. Contractor RSIS]

I tested this with kernel version 2.2.18 and arp_filter appeared to be
broken... I enabled it for /proc/sys/net/ipv4/conf/all/arp_filter,
/proc/sys/net/ipv4/conf/eth0/arp_filter and
/proc/sys/net/ipv4/conf/eth1/arp_filter and it did not change the arp
behavior at all.  I enabled hidden and it worked, is there a know problem
with this functionality?

	Sam Bingner
	PACAF CSS/SCHE
	Contractor RSIS
	DSN	315 449-7889
	COMM	808 449-7889


-----Original Message-----
From: Christopher Friesen [mailto:cfriesen@nortelnetworks.com]
Sent: Tuesday, April 17, 2001 4:25 AM
To: Sampsa Ranta
Cc: linux-net; linux-kernel
Subject: Re: ARP responses broken!


Sampsa Ranta wrote:

> I have two interfaces that share same subnet, I call eth0 194.29.192.37
> and eth1 194.29.192.38. I have forwarding turned on, proxy arp is not
> neighter are redirects.
> 
> When I flush local neighbor table in other machine I use to observe the
> response and ping the router I get response like:
> 
> 23:38:25.278848 > arp who-has 194.29.192.38 tell 194.29.192.10
(0:50:da:82:ae:9f)
> 23:38:25.278988 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:64
(0:50:da:82:ae:9f)
> 23:38:25.279009 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:6c
(0:50:da:82:ae:9f)
> 
> The second one is the valid one, but both interfaces seem to answer to the
> broadcasted packet with their own ARP addresses.

This is the default Linux behaviour.  It can be turned off by running the
following command as root:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

This ensures that interfaces will only respond to arp requests for IP
addresses
which are configured as belonging to that particular interface.

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: ARP responses broken!

[Sam.Bingner]

Correction, that was on kernel v2.2.19

Sam

****** Forwarded Message Follows *******
>To: "'Christopher Friesen'" <cfriesen@nortelnetworks.com>,   Sampsa Ranta	

<sampsa@netsonic.fi>
>From: "Bingner Sam J. Contractor RSIS" <Sam.Bingner@hickam.af.mil>
>Date: Tue, 17 Apr 2001 18:07:41 -0000
>
>I tested this with kernel version 2.2.18 and arp_filter appeared to be
>broken... I enabled it for /proc/sys/net/ipv4/conf/all/arp_filter,
>/proc/sys/net/ipv4/conf/eth0/arp_filter and
>/proc/sys/net/ipv4/conf/eth1/arp_filter and it did not change the arp
>behavior at all.  I enabled hidden and it worked, is there a know problem
>with this functionality?
>
>	Sam Bingner
>	PACAF CSS/SCHE
>	Contractor RSIS
>	DSN	315 449-7889
>	COMM	808 449-7889
>

Re: ARP responses broken!

[Rogier Wolff]

Alan Cox wrote:
> > I was asking because I had this problem before (router with two cards
> > against one physical subnet) and arpwatch complained that the router kept
> > switching MACaddresses all the time.
 
> That sounds like a bug in arpwatch. A box can have multiple mac
> addresses. Its probably a tricky one to handle but arpwatch I guess
> should spot and cope with repeated transitions between the same set
> of addresses as one warning

Well, two. Or three. 

- Hey, IP x changed from mac X to mac Y. 
- Hey, IP x changed back again to X. 
- Hmm. IP X seems to be using both Mac X and and Mac Y. 
	No further warnings will be issued about this. 

If someone is taking over an IP address (which is especially what
arpwatch should be looking for), this is exactly what you'll see. Having
the issue be ignored after one warning is bad. 

Oh, and I know people who swear that this would be an invalid
configuration, so that it is good for arpwatch to should loud and
clear about it...

				Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots. 

Re: ARP responses broken!

[Julian Anastasov]

	Hello,

Sampsa Ranta wrote:

> 23:38:25.278848 > arp who-has 194.29.192.38 tell 194.29.192.10 (0:50:da:82:ae:9f)
> 23:38:25.278988 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:64 (0:50:da:82:ae:9f)
> 23:38:25.279009 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:6c (0:50:da:82:ae:9f)
>
> The second one is the valid one, but both interfaces seem to answer to the
> broadcasted packet with their own ARP addresses.

	arp_filter is not broken, it is simply not for your setup.
It depends what you want to achieve by defining two IP addresses in
different interfaces. Considering the fact you have two addresses
in one subnet you need the incoming traffic to come from the two
interfaces. In this case you need "hidden". For the outgoing traffic:
it is controlled only from the routing.

	While in your setup arp_filter and rp_filter will ARP answer
from one card only, for the both addresses, hidden will answer from the
both cards, "correctly" in your eyes. Use arp_filter for different
nets only, i.e. when the ARP probes come from different nets in your
routing universe. hidden does not depend on nets/subnets. But may
be there are exceptions I'm missing and the other guys can correct me.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: ARP responses broken!

[Sampsa Ranta]

On Wed, 18 Apr 2001, Julian Anastasov wrote:

>
> 	Hello,
>
> Sampsa Ranta wrote:
>
> > 23:38:25.278848 > arp who-has 194.29.192.38 tell 194.29.192.10 (0:50:da:82:ae:9f)
> > 23:38:25.278988 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:64 (0:50:da:82:ae:9f)
> > 23:38:25.279009 < arp reply 194.29.192.38 is-at 0:1:2:dc:d2:6c (0:50:da:82:ae:9f)
> >
> > The second one is the valid one, but both interfaces seem to answer to the
> > broadcasted packet with their own ARP addresses.
>
> 	arp_filter is not broken, it is simply not for your setup.
> It depends what you want to achieve by defining two IP addresses in
> different interfaces. Considering the fact you have two addresses
> in one subnet you need the incoming traffic to come from the two
> interfaces. In this case you need "hidden". For the outgoing traffic:
> it is controlled only from the routing.
>
> 	While in your setup arp_filter and rp_filter will ARP answer
> from one card only, for the both addresses, hidden will answer from the
> both cards, "correctly" in your eyes. Use arp_filter for different
> nets only, i.e. when the ARP probes come from different nets in your
> routing universe. hidden does not depend on nets/subnets. But may
> be there are exceptions I'm missing and the other guys can correct me.

Yes, I wan't that other routers only see the MAC address of the interface
I assigned the IP address for if someone asks it by ARP. I also control
outgoing traffic with routing. But how am I supposed to do this in 2.4
enviroment?

I also would want outgoing traffic to work from either of the interfaces,
not depending from which the incoming traffic did come, so outgoing ARP
requests would have the MAC address and IP address of the interface they
were asked from. For filtering IP traffic, the iptables is good enough so
I can filter just as I want, I don't need filtering to happen by routing
table at the moment.

Yeah, and I am aware that routing protocols won't make this very much
easier but OSPF for example communicates only via one interface per subnet
at a time so I can use the other for my static configuration. Zebra
also wants the packets to come from the interface it expects them to, and
that is the one it is assigned.

This is what my patch was supposed to deal with, but there was something
missing from it, right?

As far as I consider routing, the routers only determinate next-hop,
routing thinks the next-hop is IP. But actually the IP number given to
routing is nothing more than a key that is passed to ARP layer to
determinate the layer 2 address. And when I don't want to make stupid
subnets when I actually can work under one network, I would like to be
able to "mark" interface by giving it IP and when I point something to
that specific IP from other host, I would like to have the traffic to be
go via that interface. If I wanted it to go via two interfaces, I would
need to deal with load balancing issues on both ends, randomly taken
interface from ARP layer is nothing I call load balancing, and the
interface could even be the same to all.

Yeah, I could do this by defining ARP addresses by hand but I would rather
not do that because there are two many routers involved.

Btw. Kernel sets up two routes to the subnet, which one is selected and
     should the other one be deleted?

Thanks,
   Sampsa Ranta
   sampsa@netsonic.fi

Re: ARP responses broken!

[Julian Anastasov]

	Hello,

On Thu, 19 Apr 2001, Sampsa Ranta wrote:

> Yes, I wan't that other routers only see the MAC address of the interface
> I assigned the IP address for if someone asks it by ARP. I also control
> outgoing traffic with routing. But how am I supposed to do this in 2.4
> enviroment?

	Eric Weigle exactly pointed out the solutions and the links.
There is still no standard support in the 2.4 kernel for your problem
but you can try http://www.linuxvirtualserver.org/hidden-2.3.41-1.diff
You can see some fails while applying but don't worry. You can try
the same 2.4.2 patch (I should make actual diff one day, for now may be
I'm trying to determine how long can live this patch) from
http://marc.theaimsgroup.com/?l=linux-virtual-server&m=98286657511673&w=2
This functionality is same as in Linux 2.2

> I also would want outgoing traffic to work from either of the interfaces,
> not depending from which the incoming traffic did come, so outgoing ARP
> requests would have the MAC address and IP address of the interface they
> were asked from. For filtering IP traffic, the iptables is good enough so

	With "hidden" you will see that your ARP probes can go through
any device without problem, of course, if the targets listen everywhere.
If an address can't be announced from the outgoing device, it is replaced
with valid source address in the ARP probe. You still can need some
source routing rules if you want the other IP traffic with specific
source address to go through specific interface. In other case (routes
by destination, for example), your ARP probes and the IP traffic will
go through one device only.

> I can filter just as I want, I don't need filtering to happen by routing
> table at the moment.

	Fine :) Because the "hidden" usage assumes secure environment
in some cases.

> Yeah, I could do this by defining ARP addresses by hand but I would rather
> not do that because there are two many routers involved.

        The ARP usage is recommended :)

> Btw. Kernel sets up two routes to the subnet, which one is selected and
>      should the other one be deleted?

        You can determine it with:

ip route get DESTINATION_IP

	Only the displayed route will be selected but you don't need
to delete the other.

> Thanks,
>    Sampsa Ranta
>    sampsa@netsonic.fi


Regards

--
Julian Anastasov <ja@ssi.bg>