[Lustre-devel] extend lnet_notify to public LNet API

[Lustre-devel] extend lnet_notify to public LNet API

[Nic Henke]

We'd like to allow upper layers (Lustre, Cray DVS, etc) to register a 
callback that would be called from lnet_notify. This will allow them to 
be notified when the lower layers have seen network problems between 
NIDs and let them take appropriate action. The upper layer could also be 
notified when that peer has returned to 'network health' after the LND 
gets its act together.

This would help allow upper layers to aggressively resend/reconnect in 
the cases where all TX have completed successfully (meaning no LNet -EIO 
on LND errors) but there are LNET_MSG_ACK or other REPLY traffic 
outstanding.

Initial proposal is on the verbose side, giving all data that 
lnet_notify sees:
- lnet_nid_t
- is_alive (boolean)
- cfs_time_t when (unsigned long on Linux) - jiffies when last alive

Is this workable and likely to be accepted up-stream ?

Nic

[Lustre-devel] extend lnet_notify to public LNet API

[liang Zhen]

Nic,

Are you suggesting to provide a new API like:

int LNetNotificationAttach(lnet_notification_callback_t callback);

to register a global callback for LNet, the callback will be called on 
any lnet_notify_locked? If so I don't see any reason we can't do this, 
at least from my point of view. One thing we need to concern is that we 
can't get such a notification for remote peers because no direct 
connection with them in LNDs, we can only get notification for routers 
but upper layer wouldn't be so interested in routers.

Also, seems to me it's a much bigger change in upper layer than in LNet.

Regards
Liang

On 11/17/10 12:00 AM, Nic Henke wrote:
> We'd like to allow upper layers (Lustre, Cray DVS, etc) to register a
> callback that would be called from lnet_notify. This will allow them to
> be notified when the lower layers have seen network problems between
> NIDs and let them take appropriate action. The upper layer could also be
> notified when that peer has returned to 'network health' after the LND
> gets its act together.
>
> This would help allow upper layers to aggressively resend/reconnect in
> the cases where all TX have completed successfully (meaning no LNet -EIO
> on LND errors) but there are LNET_MSG_ACK or other REPLY traffic
> outstanding.
>
> Initial proposal is on the verbose side, giving all data that
> lnet_notify sees:
> - lnet_nid_t
> - is_alive (boolean)
> - cfs_time_t when (unsigned long on Linux) - jiffies when last alive
>
> Is this workable and likely to be accepted up-stream ?
>
> Nic
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] extend lnet_notify to public LNet API

[Alexey Lyashkov]

Nic,

that idea discussed some time ago (as i remember with green and maxim), but have some objection.
Currently LNet hide from ptlrpc layer any network flaps, and LNet will resend request without notify ptlrpc about flap until ptlrpc request timeout.
But if ptlrpc will see node down event, ptlrpc will try reconnect  - that will produce extra overhead, because need to resend too much requests from sending and delay lists instead of lots requests in network flap time.
So, you need separate network flap from node down situation - before implementing that.
currently node marked down if node don't respond for request in ptlrpc timeout, which include network transmit and processing times, but it different then LNet message timeout.


On Nov 16, 2010, at 19:00, Nic Henke wrote:

> We'd like to allow upper layers (Lustre, Cray DVS, etc) to register a 
> callback that would be called from lnet_notify. This will allow them to 
> be notified when the lower layers have seen network problems between 
> NIDs and let them take appropriate action. The upper layer could also be 
> notified when that peer has returned to 'network health' after the LND 
> gets its act together.
> 
> This would help allow upper layers to aggressively resend/reconnect in 
> the cases where all TX have completed successfully (meaning no LNet -EIO 
> on LND errors) but there are LNET_MSG_ACK or other REPLY traffic 
> outstanding.
> 
> Initial proposal is on the verbose side, giving all data that 
> lnet_notify sees:
> - lnet_nid_t
> - is_alive (boolean)
> - cfs_time_t when (unsigned long on Linux) - jiffies when last alive
> 
> Is this workable and likely to be accepted up-stream ?
> 
> Nic
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] extend lnet_notify to public LNet API

[Nic Henke]

On 11/17/2010 01:52 AM, Alexey Lyashkov wrote:
> Nic,
>
> that idea discussed some time ago (as i remember with green and maxim), but have some objection.
> Currently LNet hide from ptlrpc layer any network flaps, and LNet will resend request without notify ptlrpc about flap until ptlrpc request timeout.

I'm missing something - to my knowledge, LNet never retries messages.

> But if ptlrpc will see node down event, ptlrpc will try reconnect  - that will produce extra overhead, because need to resend too much requests from sending and delay lists instead of lots requests in network flap time.
> So, you need separate network flap from node down situation - before implementing that.
> currently node marked down if node don't respond for request in ptlrpc timeout, which include network transmit and processing times, but it different then LNet message timeout.

I think that is a valid upper layer decision to make, but separate from 
implementing the LNet callbacks on network 'flap'. I wouldn't want to 
force ptlrpc to use it.

Cheers,
Nic

[Lustre-devel] extend lnet_notify to public LNet API

[Nic Henke]

On 11/17/2010 08:59 AM, Nic Henke wrote:
> On 11/17/2010 01:52 AM, Alexey Lyashkov wrote:
>> Nic,
>>
>> that idea discussed some time ago (as i remember with green and maxim), but have some objection.
>> Currently LNet hide from ptlrpc layer any network flaps, and LNet will resend request without notify ptlrpc about flap until ptlrpc request timeout.
>
> I'm missing something - to my knowledge, LNet never retries messages.
>
>> But if ptlrpc will see node down event, ptlrpc will try reconnect  - that will produce extra overhead, because need to resend too much requests from sending and delay lists instead of lots requests in network flap time.
>> So, you need separate network flap from node down situation - before implementing that.
>> currently node marked down if node don't respond for request in ptlrpc timeout, which include network transmit and processing times, but it different then LNet message timeout.
>
> I think that is a valid upper layer decision to make, but separate from
> implementing the LNet callbacks on network 'flap'. I wouldn't want to
> force ptlrpc to use it.

Any response to this ?

Nic

[Lustre-devel] extend lnet_notify to public LNet API

[Nic Henke]

On 11/16/2010 10:00 AM, Nic Henke wrote:
> We'd like to allow upper layers (Lustre, Cray DVS, etc) to register a
> callback that would be called from lnet_notify. This will allow them to
> be notified when the lower layers have seen network problems between
> NIDs and let them take appropriate action. The upper layer could also be
> notified when that peer has returned to 'network health' after the LND
> gets its act together.
>
> This would help allow upper layers to aggressively resend/reconnect in
> the cases where all TX have completed successfully (meaning no LNet -EIO
> on LND errors) but there are LNET_MSG_ACK or other REPLY traffic
> outstanding.
>
> Initial proposal is on the verbose side, giving all data that
> lnet_notify sees:
> - lnet_nid_t
> - is_alive (boolean)
> - cfs_time_t when (unsigned long on Linux) - jiffies when last alive
>

One oddity - if the LND has peer_health disabled (no ni_peertimeout 
value), there doesn't seem to be anything that'd set the peer back to 
'up'. Am I missing something or is this as desired ?

Nic

[Lustre-devel] extend lnet_notify to public LNet API

[Eric Barton]

Nic,

There's no fundamental reason we wouldn't want to add the ability to
register callbacks to be called from lnet_notify().  But we should
take care of the following details...

0. You will never receive notifications about NIDs that you are not on
   your local LNDs.  For example, if you're a client on a routed
   network, you'll only find out about router health, never about
   server health.

1. Don't just assume only 1 callback can be registered - if this is
   generally useful, several LNET users might want such notification
   and so the API and implementation should allow for it.

2. lnet_notify_locked() is where all notifications arrive.  It's
   called (as the name suggests) holding LNET locks, so any callback
   handling must complete in very well bounded time just like event
   callbacks.

Shadow wrote...

> that idea discussed some time ago (as i remember with green and
> maxim), but have some objection.  Currently LNet hide from ptlrpc
> layer any network flaps, and LNet will resend request without notify
> ptlrpc about flap until ptlrpc request timeout.  

LNET does not resend.  LNet itself doesn't know about "network flaps"
- all it has to go on is whether communications with individual peer
nodes succeed or fail.  

Isaac (correct me if this is too broad brush) most recently worked on
fixing some pathalogical issues with dead peers to ensure that
communications with a _known_ dead peer complete quickly with failure.
Now, the only time communications can be blocked for the whole LND
timeout should be on initial connection establishment to a peer in an
unknown state, or at the point that the peer dies.

> But if ptlrpc will see node down event, ptlrpc will try reconnect -
> that will produce extra overhead, because need to resend too much
> requests from sending and delay lists instead of lots requests in
> network flap time.  

Ptlrpc reconnect to a know dead peer will fail immediately.  However
this communication attempt will cause the LND to attempt connection
re-establishment and if this is successful, LNET will mark the peer
alive.  Subsequent communication attempts should now succeed.

> So, you need separate network flap from node down situation - before
> implementing that.  currently node marked down if node don't respond
> for request in ptlrpc timeout, which include network transmit and
> processing times, but it different then LNet message timeout.

Indeed.  This is further complicated by LNET routing.  Communications
buffered in a router are lost when the router fails, so you have to
regard routed LNETs as potentially lossy.

Nic wrote...

> One oddity - if the LND has peer_health disabled (no ni_peertimeout 
> value), there doesn't seem to be anything that'd set the peer back to 
> 'up'. Am I missing something or is this as desired ?

Hmmm - I originally implemented peer aliveness only for LNET routers
to ensure known dead routers were avoided - so the status of
non-router peers was not used or maintained properly (mea maxima
culpa).

Isaac fixed this more generally as I mentioned above, but it looks
like only socklnd and o2iblnd have the support.  Shouldn't be too hard
to add for ptllnd though :)

-- 

                Cheers,
                        Eric