Re: [PATCH net] nfp: flower: cmsg rtnl locks can timeout reify messages

[David Miller <davem@davemloft.net>]

From: Simon Horman <simon.horman@netronome.com>
Date: Fri,  6 Sep 2019 19:29:41 +0200

> From: Fred Lotter <frederik.lotter@netronome.com>
> 
> Flower control message replies are handled in different locations. The truly
> high priority replies are handled in the BH (tasklet) context, while the
> remaining replies are handled in a predefined Linux work queue. The work
> queue handler orders replies into high and low priority groups, and always
> start servicing the high priority replies within the received batch first.
> 
> Reply Type:			Rtnl Lock:	Handler:
 ...
> A subset of control messages can block waiting for an rtnl lock (from both
> work queue priority groups). The rtnl lock is heavily contended for by
> external processes such as systemd-udevd, systemd-network and libvirtd,
> especially during netdev creation, such as when flower VFs and representors
> are instantiated.
> 
> Kernel netlink instrumentation shows that external processes (such as
> systemd-udevd) often use successive rtnl_trylock() sequences, which can result
> in an rtnl_lock() blocked control message to starve for longer periods of time
> during rtnl lock contention, i.e. netdev creation.
> 
> In the current design a single blocked control message will block the entire
> work queue (both priorities), and introduce a latency which is
> nondeterministic and dependent on system wide rtnl lock usage.
> 
> In some extreme cases, one blocked control message at exactly the wrong time,
> just before the maximum number of VFs are instantiated, can block the work
> queue for long enough to prevent VF representor REIFY replies from getting
> handled in time for the 40ms timeout.
> 
> The firmware will deliver the total maximum number of REIFY message replies in
> around 300us.
> 
> Only REIFY and MTU update messages require replies within a timeout period (of
> 40ms). The MTU-only updates are already done directly in the BH (tasklet)
> handler.
> 
> Move the REIFY handler down into the BH (tasklet) in order to resolve timeouts
> caused by a blocked work queue waiting on rtnl locks.
> 
> Signed-off-by: Fred Lotter <frederik.lotter@netronome.com>
> Signed-off-by: Simon Horman <simon.horman@netronome.com>

Applied.