How to make stack send broadcast ARP request when entry is STALE?

How to make stack send broadcast ARP request when entry is STALE?

[Ulf samuelsson]

Have a problem with an HP router at a certain location, which
is configured to only answer to broadcast ARP requests.
That cannot be changed.

The first ARP request the kernel sends out, is a broadcast request,
which is fine, but after the reply, the kernel sends unicast requests,
which will not get any replies.

The ARP entry will after some time enter STALE state,
and if nothing is done it will time out, and be removed.
This process takes to long, and I have been told that it is
difficult to makes changes that will eventually remove it.

Have tried to change the state from STALE to INCOMPLETE, which failed,
and then tried to change the state to PROBE which also failed.

The stack is only sending out unicasts, and never broadcast.
Is there any way to get the stack to send out a broadcast ARP
without having to wait for the entry to be removed?

I think the recommended behaviour in IPv6 is to send out 3 unicasts
and if all fails, to send out broadcasts.

Anyone know any good literature on how the ARP + neigh state machine works
in the kernel.

I read in Herberts book about the Linux TCP/IP stack and it only discuss how to reply to 
ARP requests and not anything on how to generate ARP requests.

Best Regards
Ulf Samuelsson

Re: How to make stack send broadcast ARP request when entry is STALE?

[Sowmini Varadhan]

On Wed, Nov 5, 2014 at 1:48 AM, Ulf samuelsson <netdev@emagii.com> wrote:
> Have a problem with an HP router at a certain location, which
> is configured to only answer to broadcast ARP requests.
> That cannot be changed.
>
> The first ARP request the kernel sends out, is a broadcast request,
> which is fine, but after the reply, the kernel sends unicast requests,
> which will not get any replies.

I'm afraid that's a problem with your HP router. There is nothing
in RFC 826 that says that the ARP request MUST be broadcast,
and many OS'es  will follow the IPv6 ND model of sending
unicast ARP requests to re-verify cached arp information.
Looks like the "kernel" you refer to above is following this
state machine, which is a good and healthy thing- you dont
want to be needlessly broadcasting arp reverification requests.

thus even if you have access to the kernel source code (as with linux)
it's not recommended to broadcast the ARP reverification.

> The ARP entry will after some time enter STALE state,
> and if nothing is done it will time out, and be removed.
 :

> I think the recommended behaviour in IPv6 is to send out 3 unicasts
> and if all fails, to send out broadcasts.

Be careful about the distinction between IPv4 ARP and IPv6 ND.
The base state-machine for IPv6 is explicitly specified in RFC 4861 (with
ongoing work to optimize some of this in various ietf working groups)

The territory is less clearly defined for ARP. The base protocol comes from
RFC 726, which does not address DAD  at all- that only comes from rfc 5227.
Some OS-es (solaris, microsoft?) implelement the same state-machine
for both IPv4 and IPv6, while also support 5227.

>
> Anyone know any good literature on how the ARP + neigh state machine works
> in the kernel.
>
> I read in Herberts book about the Linux TCP/IP stack and it only discuss how to reply to
> ARP requests and not anything on how to generate ARP requests.

Hope that helps.

--Sowmini

Re: How to make stack send broadcast ARP request when entry is STALE?

[Ulf samuelsson]

Best Regards
Ulf Samuelsson
ulf@emagii.com
+46  (722) 427 437


> 6 nov 2014 kl. 12:48 skrev Sowmini Varadhan <sowmini05@gmail.com>:
> 
>> On Wed, Nov 5, 2014 at 1:48 AM, Ulf samuelsson <netdev@emagii.com> wrote:
>> Have a problem with an HP router at a certain location, which
>> is configured to only answer to broadcast ARP requests.
>> That cannot be changed.
>> 
>> The first ARP request the kernel sends out, is a broadcast request,
>> which is fine, but after the reply, the kernel sends unicast requests,
>> which will not get any replies.
> 
> I'm afraid that's a problem with your HP router. There is nothing
> in RFC 826 that says that the ARP request MUST be broadcast,
> and many OS'es  will follow the IPv6 ND model of sending
> unicast ARP requests to re-verify cached arp information.
> Looks like the "kernel" you refer to above is following this
> state machine, which is a good and healthy thing- you dont
> want to be needlessly broadcasting arp reverification requests.
> 

The router setup will not be changed, not my decision. i have to live with it.
The issue occurs only on a backplane, implemented using network,
so consequences of broadcast ARP if unicast ARP fails is known, and acceptable,

Since the controlling mechanism is a kernel module
it would make sense to use the native stack, instead of trying to duplicate the functionality.

If I dont want any broadcast ARP, there is a mcast_probes /proc variable that can be changed
to zero (default 3).

As it is now, it look like mcast_probes is there, but it is not used at all by the kernel,
so it should either be removed, or the intended functionality should work

It is strange, that if even if it is acceptable with broadcast ARP,
you cannot configure the kernel code, nor the kernel runtime
to generate it.

I think the kernel should be changed to have a compile time option
allowing broadcast ARP to be sent, if mcast_probes > 0.
Alternatively a new variable should be introduced which will allow sending broadcast ARP
if the unicast ARP fails,

Will implement that fix locally anyway.

> thus even if you have access to the kernel source code (as with linux)
> it's not recommended to broadcast the ARP reverification.
> 
>> The ARP entry will after some time enter STALE state,
>> and if nothing is done it will time out, and be removed.
> :
> 
>> I think the recommended behaviour in IPv6 is to send out 3 unicasts
>> and if all fails, to send out broadcasts.
> 
> Be careful about the distinction between IPv4 ARP and IPv6 ND.
> The base state-machine for IPv6 is explicitly specified in RFC 4861 (with
> ongoing work to optimize some of this in various ietf working groups)
> 
> The territory is less clearly defined for ARP. The base protocol comes from
> RFC 726, which does not address DAD  at all- that only comes from rfc 5227.
> Some OS-es (solaris, microsoft?) implelement the same state-machine
> for both IPv4 and IPv6, while also support 5227.
> 
>> 
>> Anyone know any good literature on how the ARP + neigh state machine works
>> in the kernel.
>> 
>> I read in Herberts book about the Linux TCP/IP stack and it only discuss how to reply to
>> ARP requests and not anything on how to generate ARP requests.
> 
> Hope that helps.
> 
> --Sowmini
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to make stack send broadcast ARP request when entry is STALE?

[Brian Haley]

On 11/05/2014 07:48 AM, Ulf samuelsson wrote:
> Have a problem with an HP router at a certain location, which
> is configured to only answer to broadcast ARP requests.
> That cannot be changed.

Sorry to hear about the problem, but my only suggestions would be to try 
the latest firmware and/or put a call in to support.  I don't happen 
work in the division that makes routers...

> The first ARP request the kernel sends out, is a broadcast request,
> which is fine, but after the reply, the kernel sends unicast requests,
> which will not get any replies.

You might be able to hack this by inserting an ebtables rule - check the 
dnat target section of the man page - don't know the exact syntax but it 
would probably end in '-j dnat --to-destination ff:ff:ff:ff:ff:ff'

-Brian

Re: How to make stack send broadcast ARP request when entry is STALE?

[Ulf samuelsson]

The HP router is configured by a customer, and they intentionally limit replies
to broadcast, and that is how they want it.

In the previous version of the build system, the Interpeak stack was used
and this would in PROBE state send unicast ARP request, and if that failed
send broadcast ARP.

The native linux stack, when in PROBE state sends only unicast until it decides
that it should enter FAILED state.

The 'mcast_probes' variable seems to be totally ignored, except the first  time,
so I do not see why it is there.

Best Regards
Ulf Samuelsson
ulf@emagii.com
+46  (722) 427 437


> 7 nov 2014 kl. 10:54 skrev Brian Haley <brian.haley@hp.com>:
> 
>> On 11/05/2014 07:48 AM, Ulf samuelsson wrote:
>> Have a problem with an HP router at a certain location, which
>> is configured to only answer to broadcast ARP requests.
>> That cannot be changed.
> 
> Sorry to hear about the problem, but my only suggestions would be to try the latest firmware and/or put a call in to support.  I don't happen work in the division that makes routers...
> 
>> The first ARP request the kernel sends out, is a broadcast request,
>> which is fine, but after the reply, the kernel sends unicast requests,
>> which will not get any replies.
> 
> You might be able to hack this by inserting an ebtables rule - check the dnat target section of the man page - don't know the exact syntax but it would probably end in '-j dnat --to-destination ff:ff:ff:ff:ff:ff'
> 
> -Brian
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to make stack send broadcast ARP request when entry is STALE?

[Brian Haley]

On 11/07/2014 05:11 AM, Ulf samuelsson wrote:
> The HP router is configured by a customer, and they intentionally limit replies
> to broadcast, and that is how they want it.

So this is the crux of the problem - the customer has configured the router so
that it doesn't play well with most modern network stacks that try and use
unicast so they don't send unnecessary broadcast packets.  I don't know why I
thought this was something wrong with the router software.

Did you try this?

$ sudo sysctl net.ipv4.neigh.eth0.ucast_solicit=0

It works for me.

And they really should re-think their decision on that configuration setting.

-Brian


> In the previous version of the build system, the Interpeak stack was used
> and this would in PROBE state send unicast ARP request, and if that failed
> send broadcast ARP.
> 
> The native linux stack, when in PROBE state sends only unicast until it decides
> that it should enter FAILED state.
> 
> The 'mcast_probes' variable seems to be totally ignored, except the first  time,
> so I do not see why it is there.
> 
> Best Regards
> Ulf Samuelsson
> ulf@emagii.com
> +46  (722) 427 437
> 
> 
>> 7 nov 2014 kl. 10:54 skrev Brian Haley <brian.haley@hp.com>:
>>
>>> On 11/05/2014 07:48 AM, Ulf samuelsson wrote:
>>> Have a problem with an HP router at a certain location, which
>>> is configured to only answer to broadcast ARP requests.
>>> That cannot be changed.
>>
>> Sorry to hear about the problem, but my only suggestions would be to try the latest firmware and/or put a call in to support.  I don't happen work in the division that makes routers...
>>
>>> The first ARP request the kernel sends out, is a broadcast request,
>>> which is fine, but after the reply, the kernel sends unicast requests,
>>> which will not get any replies.
>>
>> You might be able to hack this by inserting an ebtables rule - check the dnat target section of the man page - don't know the exact syntax but it would probably end in '-j dnat --to-destination ff:ff:ff:ff:ff:ff'
>>
>> -Brian
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: How to make stack send broadcast ARP request when entry is STALE?

[Ulf samuelsson]

If I set ucast_solicit to '0', then I always send broadcast, which is not desirable.

In the PROBE state of the ARP state machine, "probes" count from 0 .. ucast_probes.

I can see the following arguments for letting "probes" count from 
  
   0.. (ucast_probes + app_probes + mcast_probes)
   

A: This is how the IPv6 is doing it. 
     It is not standardized in IPv4, but why should the IPv4 have a different behaviour?

B: If you do not send out broadcast if unicast fails, then a broadcast will be sent out 
     anyway, once the ARP entry is removed by the garbage collector.
     You get an annoyingly long delay before that happens.

C: If a large data centre does not want broadcasts to be sent out, 
     then they can set mcast_probes to 0, in which case no broadcasts will be sent
     out in PROBE state.

D:  When in other states, the counter runs until it a reply is had, or the counter wraps around.
      It is sending broadcast all the time.


Best Regards
Ulf Samuelsson
ulf@emagii.com
+46  (722) 427 437


> 10 nov 2014 kl. 23:52 skrev Brian Haley <brian.haley@hp.com>:
> 
>> On 11/07/2014 05:11 AM, Ulf samuelsson wrote:
>> The HP router is configured by a customer, and they intentionally limit replies
>> to broadcast, and that is how they want it.
> 
> So this is the crux of the problem - the customer has configured the router so
> that it doesn't play well with most modern network stacks that try and use
> unicast so they don't send unnecessary broadcast packets.  I don't know why I
> thought this was something wrong with the router software.
> 
> Did you try this?
> 
> $ sudo sysctl net.ipv4.neigh.eth0.ucast_solicit=0
> 
> It works for me.
> 
> And they really should re-think their decision on that configuration setting.
> 
> -Brian
> 
> 
>> In the previous version of the build system, the Interpeak stack was used
>> and this would in PROBE state send unicast ARP request, and if that failed
>> send broadcast ARP.
>> 
>> The native linux stack, when in PROBE state sends only unicast until it decides
>> that it should enter FAILED state.
>> 
>> The 'mcast_probes' variable seems to be totally ignored, except the first  time,
>> so I do not see why it is there.
>> 
>> Best Regards
>> Ulf Samuelsson
>> ulf@emagii.com
>> +46  (722) 427 437
>> 
>> 
>>>> 7 nov 2014 kl. 10:54 skrev Brian Haley <brian.haley@hp.com>:
>>>> 
>>>> On 11/05/2014 07:48 AM, Ulf samuelsson wrote:
>>>> Have a problem with an HP router at a certain location, which
>>>> is configured to only answer to broadcast ARP requests.
>>>> That cannot be changed.
>>> 
>>> Sorry to hear about the problem, but my only suggestions would be to try the latest firmware and/or put a call in to support.  I don't happen work in the division that makes routers...
>>> 
>>>> The first ARP request the kernel sends out, is a broadcast request,
>>>> which is fine, but after the reply, the kernel sends unicast requests,
>>>> which will not get any replies.
>>> 
>>> You might be able to hack this by inserting an ebtables rule - check the dnat target section of the man page - don't know the exact syntax but it would probably end in '-j dnat --to-destination ff:ff:ff:ff:ff:ff'
>>> 
>>> -Brian
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
> 

Re: How to make stack send broadcast ARP request when entry is STALE?

[Brian Haley]

On 11/11/2014 05:08 AM, Ulf samuelsson wrote:
> If I set ucast_solicit to '0', then I always send broadcast, which is not desirable.

That seems to contradict your statement:

"The HP router is configured by a customer, and they intentionally limit replies
 to broadcast, and that is how they want it."

So I'm not understanding your question.

The best way forward would be for you to send a patch out that gets your desired
behaviour, and let others give feedback on it.

-Brian

> In the PROBE state of the ARP state machine, "probes" count from 0 .. ucast_probes.
> 
> I can see the following arguments for letting "probes" count from 
>   
>    0.. (ucast_probes + app_probes + mcast_probes)
>    
> 
> A: This is how the IPv6 is doing it. 
>      It is not standardized in IPv4, but why should the IPv4 have a different behaviour?
> 
> B: If you do not send out broadcast if unicast fails, then a broadcast will be sent out 
>      anyway, once the ARP entry is removed by the garbage collector.
>      You get an annoyingly long delay before that happens.
> 
> C: If a large data centre does not want broadcasts to be sent out, 
>      then they can set mcast_probes to 0, in which case no broadcasts will be sent
>      out in PROBE state.
> 
> D:  When in other states, the counter runs until it a reply is had, or the counter wraps around.
>       It is sending broadcast all the time.
> 
> 
> Best Regards
> Ulf Samuelsson
> ulf@emagii.com
> +46  (722) 427 437
> 
> 
>> 10 nov 2014 kl. 23:52 skrev Brian Haley <brian.haley@hp.com>:
>>
>>> On 11/07/2014 05:11 AM, Ulf samuelsson wrote:
>>> The HP router is configured by a customer, and they intentionally limit replies
>>> to broadcast, and that is how they want it.
>>
>> So this is the crux of the problem - the customer has configured the router so
>> that it doesn't play well with most modern network stacks that try and use
>> unicast so they don't send unnecessary broadcast packets.  I don't know why I
>> thought this was something wrong with the router software.
>>
>> Did you try this?
>>
>> $ sudo sysctl net.ipv4.neigh.eth0.ucast_solicit=0
>>
>> It works for me.
>>
>> And they really should re-think their decision on that configuration setting.
>>
>> -Brian
>>
>>
>>> In the previous version of the build system, the Interpeak stack was used
>>> and this would in PROBE state send unicast ARP request, and if that failed
>>> send broadcast ARP.
>>>
>>> The native linux stack, when in PROBE state sends only unicast until it decides
>>> that it should enter FAILED state.
>>>
>>> The 'mcast_probes' variable seems to be totally ignored, except the first  time,
>>> so I do not see why it is there.
>>>
>>> Best Regards
>>> Ulf Samuelsson
>>> ulf@emagii.com
>>> +46  (722) 427 437
>>>
>>>
>>>>> 7 nov 2014 kl. 10:54 skrev Brian Haley <brian.haley@hp.com>:
>>>>>
>>>>> On 11/05/2014 07:48 AM, Ulf samuelsson wrote:
>>>>> Have a problem with an HP router at a certain location, which
>>>>> is configured to only answer to broadcast ARP requests.
>>>>> That cannot be changed.
>>>>
>>>> Sorry to hear about the problem, but my only suggestions would be to try the latest firmware and/or put a call in to support.  I don't happen work in the division that makes routers...
>>>>
>>>>> The first ARP request the kernel sends out, is a broadcast request,
>>>>> which is fine, but after the reply, the kernel sends unicast requests,
>>>>> which will not get any replies.
>>>>
>>>> You might be able to hack this by inserting an ebtables rule - check the dnat target section of the man page - don't know the exact syntax but it would probably end in '-j dnat --to-destination ff:ff:ff:ff:ff:ff'
>>>>
>>>> -Brian
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> 

Re: How to make stack send broadcast ARP request when entry is STALE?

[Ulf samuelsson]

> 11 nov 2014 kl. 17:36 skrev Brian Haley <brian.haley@hp.com>:
> 
>> On 11/11/2014 05:08 AM, Ulf samuelsson wrote:
>> If I set ucast_solicit to '0', then I always send broadcast, which is not desirable.
> 
> That seems to contradict your statement:
> 
> "The HP router is configured by a customer, and they intentionally limit replies
> to broadcast, and that is how they want it."
> 

The equipment is beeing used in multiple locations, and this is but one.
Normally you want to follow IPv6 rules which says

* Start w broadcast
* Once reply arrives, start using unicast
* If unicast fails for 'ucast_probes', send up to 'mcast_probes' packets before FAIL.
* A reply to a broadcast packet will make the kernel revert to unicast


> So I'm not understanding your question.
> 
> The best way forward would be for you to send a patch out that gets your desired
> behaviour, and let others give feedback on it.
> 
> -Brian



Best Regards
Ulf Samuelsson



> 
>> In the PROBE state of the ARP state machine, "probes" count from 0 .. ucast_probes.
>> 
>> I can see the following arguments for letting "probes" count from 
>> 
>>   0.. (ucast_probes + app_probes + mcast_probes)
>> 
>> 
>> A: This is how the IPv6 is doing it. 
>>     It is not standardized in IPv4, but why should the IPv4 have a different behaviour?
>> 
>> B: If you do not send out broadcast if unicast fails, then a broadcast will be sent out 
>>     anyway, once the ARP entry is removed by the garbage collector.
>>     You get an annoyingly long delay before that happens.
>> 
>> C: If a large data centre does not want broadcasts to be sent out, 
>>     then they can set mcast_probes to 0, in which case no broadcasts will be sent
>>     out in PROBE state.
>> 
>> D:  When in other states, the counter runs until it a reply is had, or the counter wraps around.
>>      It is sending broadcast all the time.
>> 
>> 
>> Best Regards
>> Ulf Samuelsson
>> ulf@emagii.com
>> +46  (722) 427 437
>> 
>> 
>>>> 10 nov 2014 kl. 23:52 skrev Brian Haley <brian.haley@hp.com>:
>>>> 
>>>> On 11/07/2014 05:11 AM, Ulf samuelsson wrote:
>>>> The HP router is configured by a customer, and they intentionally limit replies
>>>> to broadcast, and that is how they want it.
>>> 
>>> So this is the crux of the problem - the customer has configured the router so
>>> that it doesn't play well with most modern network stacks that try and use
>>> unicast so they don't send unnecessary broadcast packets.  I don't know why I
>>> thought this was something wrong with the router software.
>>> 
>>> Did you try this?
>>> 
>>> $ sudo sysctl net.ipv4.neigh.eth0.ucast_solicit=0
>>> 
>>> It works for me.
>>> 
>>> And they really should re-think their decision on that configuration setting.
>>> 
>>> -Brian
>>> 
>>> 
>>>> In the previous version of the build system, the Interpeak stack was used
>>>> and this would in PROBE state send unicast ARP request, and if that failed
>>>> send broadcast ARP.
>>>> 
>>>> The native linux stack, when in PROBE state sends only unicast until it decides
>>>> that it should enter FAILED state.
>>>> 
>>>> The 'mcast_probes' variable seems to be totally ignored, except the first  time,
>>>> so I do not see why it is there.
>>>> 
>>>> Best Regards
>>>> Ulf Samuelsson
>>>> ulf@emagii.com
>>>> +46  (722) 427 437
>>>> 
>>>> 
>>>>>> 7 nov 2014 kl. 10:54 skrev Brian Haley <brian.haley@hp.com>:
>>>>>> 
>>>>>> On 11/05/2014 07:48 AM, Ulf samuelsson wrote:
>>>>>> Have a problem with an HP router at a certain location, which
>>>>>> is configured to only answer to broadcast ARP requests.
>>>>>> That cannot be changed.
>>>>> 
>>>>> Sorry to hear about the problem, but my only suggestions would be to try the latest firmware and/or put a call in to support.  I don't happen work in the division that makes routers...
>>>>> 
>>>>>> The first ARP request the kernel sends out, is a broadcast request,
>>>>>> which is fine, but after the reply, the kernel sends unicast requests,
>>>>>> which will not get any replies.
>>>>> 
>>>>> You might be able to hack this by inserting an ebtables rule - check the dnat target section of the man page - don't know the exact syntax but it would probably end in '-j dnat --to-destination ff:ff:ff:ff:ff:ff'
>>>>> 
>>>>> -Brian
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>