Problem with a host having an UNREACHABLE address ?

Problem with a host having an UNREACHABLE address ?

[Boiteux Frederic]

   Hi,

I've another problem to submit to your knowledge !

I test my SCTP messaging small application between two hosts far away
one from another, with possible packet loss on the net. Each of them is
advertising two different [IPv4] addresses, but in fact, on one host,
one IP isn't actually connected and can't be used for messages.

When I start SCTP messages exchanges, the hosts are using the first
address, which is working, and messages are correctly transmitted. On
the peer of the failing-IP host, I get notifications that this IP
address is UNREACHABLE, it's correct. In fact, I get one notification
each 100ms, but this can probably be setup.
The problem is that after some time of message exchanges, about 20
seconds, the hosts get a LOST notification for the association and no
message can't be sent anymore until a new association is built.

Should I do something when I get a UNREACHABLE peer change notification
? is the LOST related to these UNREACHABLE notifications, or should I
look for other cause ? I didn't saw this behavior during my tests (on
virtual hosts)...

                Thanks for you help,
                               Fred.

Re: Problem with a host having an UNREACHABLE address ?

[Vlad Yasevich]

On 12/03/2014 11:11 AM, Boiteux Frederic wrote:
>    Hi,
> 
> I've another problem to submit to your knowledge !
> 
> I test my SCTP messaging small application between two hosts far away
> one from another, with possible packet loss on the net. Each of them is
> advertising two different [IPv4] addresses, but in fact, on one host,
> one IP isn't actually connected and can't be used for messages.
> 
> When I start SCTP messages exchanges, the hosts are using the first
> address, which is working, and messages are correctly transmitted. On
> the peer of the failing-IP host, I get notifications that this IP
> address is UNREACHABLE, it's correct. In fact, I get one notification
> each 100ms, but this can probably be setup.

No, this was an issue that was recently addressed.  See
https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id\x061079ac0b9be7a578dcd09f7865c2c0d6ac894a

> The problem is that after some time of message exchanges, about 20
> seconds, the hosts get a LOST notification for the association and no
> message can't be sent anymore until a new association is built.

Which host gets the LOST notification?
> 
> Should I do something when I get a UNREACHABLE peer change notification
> ?

You don't have to do anything here.

> is the LOST related to these UNREACHABLE notifications, or should I
> look for other cause ?

It might be, or it might be related to some transport selection code
that has seen some good fixes.

It would be good if you could try this on a recent kernel to see if
you still see the issue.

-vlad

> I didn't saw this behavior during my tests (on
> virtual hosts)...
> 
>                 Thanks for you help,
>                                Fred.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: Problem with a host having an UNREACHABLE address ?

[Boiteux Frederic]

	Hi Vlad,

>> When I start SCTP messages exchanges, the hosts are using the first 
>> address, which is working, and messages are correctly transmitted. On 
>> the peer of the failing-IP host, I get notifications that this IP 
>> address is UNREACHABLE, it's correct. In fact, I get one notification 
>> each 100ms, but this can probably be setup.

>No, this was an issue that was recently addressed.  See >https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id\x061079ac0b9be7a578dcd09f7865c2c0d6ac894a

Right. But it's not a big problem to ignore them...

>> The problem is that after some time of message exchanges, about 20 
>> seconds, the hosts get a LOST notification for the association and no 
>> message can't be sent anymore until a new association is built.

>Which host gets the LOST notification?

It's the host receiving UNREACHABLE notifications (not the host having a failed IP, but its peer). It's why I wondered if association breakage and these notifications was related.

...

>> is the LOST related to these UNREACHABLE notifications, or should I 
>> look for other cause ?

>It might be, or it might be related to some transport selection code that has seen some good fixes.
> It would be good if you could try this on a recent kernel to see if you still see the issue.

I tested my application on virtual systems having the same kernel/system, and tried to do the same configuration, but didn't reproduce the problem.

As it's difficult/impossible to upgrade the kernel (without modifying the system, and I can't do that), do you think backporting sctp kernel module is a way to test this ?

Anyway, given the 'UNREACHABLE' notifications is a wrong suspect, I'll try to do some network packets' capture to try to understand what is going on...

  Many thanks,
             Fred.

RE: Problem with a host having an UNREACHABLE address ?

[Boiteux Frederic]

	Hello,

 I've put aside multiple receives of UNREACHABLE notifications, and done some network packets captures, and actually found my mistake : I had downsized too much the 'rto_max' parameter, for quickier detection of unavailable peers on a fast network, but here its value was like the mean roundtrip time between hosts (given by the 'ping'), and from time to time, this retransmission timeout was  exceeded ! Using a larger value for this parameter solved my problem :-)

What could be a good rto_max value calculation from the mean RTT value between hosts ?


     Thanks for your help,
		Fred.

Re: Problem with a host having an UNREACHABLE address ?

[Neil Horman]

On Wed, Dec 03, 2014 at 05:57:18PM +0100, Boiteux Frederic wrote:
> 	Hi Vlad,
> 
> >> When I start SCTP messages exchanges, the hosts are using the first 
> >> address, which is working, and messages are correctly transmitted. On 
> >> the peer of the failing-IP host, I get notifications that this IP 
> >> address is UNREACHABLE, it's correct. In fact, I get one notification 
> >> each 100ms, but this can probably be setup.
> 
> >No, this was an issue that was recently addressed.  See >https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id\x061079ac0b9be7a578dcd09f7865c2c0d6ac894a
> 
> Right. But it's not a big problem to ignore them...
> 
> >> The problem is that after some time of message exchanges, about 20 
> >> seconds, the hosts get a LOST notification for the association and no 
> >> message can't be sent anymore until a new association is built.
> 
> >Which host gets the LOST notification?
> 
> It's the host receiving UNREACHABLE notifications (not the host having a failed IP, but its peer). It's why I wondered if association breakage and these notifications was related.
> 
It sounds to me like you just can't include multihoming with the address you're
referring to.  E.g.:

Host A has ip addresses 1 and 2
Host B has ip addresses 3 and 4

Host A ip 2 isn't routeable

Regardless of Host A's unrouteable ip address 2, its getting included in the
INIT chunk when you establish a connection with Host B, and Host B is then
trying to reach Host A by ip address 2, which your network is rejecting due to
its unrouteable nature.  The only thing I'm aware of to do here is not inlucde
that address, but using the connectx call and not including the unrouteable
address.

Neil

> ...
> 
> >> is the LOST related to these UNREACHABLE notifications, or should I 
> >> look for other cause ?
> 
> >It might be, or it might be related to some transport selection code that has seen some good fixes.
> > It would be good if you could try this on a recent kernel to see if you still see the issue.
> 
> I tested my application on virtual systems having the same kernel/system, and tried to do the same configuration, but didn't reproduce the problem.
> 
> As it's difficult/impossible to upgrade the kernel (without modifying the system, and I can't do that), do you think backporting sctp kernel module is a way to test this ?
> 
> Anyway, given the 'UNREACHABLE' notifications is a wrong suspect, I'll try to do some network packets' capture to try to understand what is going on...
> 
>   Many thanks,
>              Fred.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>