Re: [patch net-next RFC 0/2] net: devlink: remove devlink big lock

[Jiri Pirko <jiri@resnulli.us>]

Wed, Jun 29, 2022 at 12:25:49PM CEST, jiri@resnulli.us wrote:
>Tue, Jun 28, 2022 at 09:43:26AM CEST, idosch@nvidia.com wrote:
>>On Mon, Jun 27, 2022 at 05:55:06PM +0200, Jiri Pirko wrote:
>>> Mon, Jun 27, 2022 at 05:41:31PM CEST, idosch@nvidia.com wrote:
>>> >On Mon, Jun 27, 2022 at 03:54:59PM +0200, Jiri Pirko wrote:
>>> >> From: Jiri Pirko <jiri@nvidia.com>
>>> >> 
>>> >> This is an attempt to remove use of devlink_mutex. This is a global lock
>>> >> taken for every user command. That causes that long operations performed
>>> >> on one devlink instance (like flash update) are blocking other
>>> >> operations on different instances.
>>> >
>>> >This patchset is supposed to prevent one devlink instance from blocking
>>> >another? Devlink does not enable "parallel_ops", which means that the
>>> >generic netlink mutex is serializing all user space operations. AFAICT,
>>> >this series does not enable "parallel_ops", so I'm not sure what
>>> >difference the removal of the devlink mutex makes.
>>> 
>>> You are correct, that is missing. For me, as a side effect this patchset
>>> resolved the deadlock for LC auxdev you pointed out. That was my
>>> motivation for this patchset :)
>>
>>Given that devlink does not enable "parallel_ops" and that the generic
>>netlink mutex is held throughout all callbacks, what prevents you from
>>simply dropping the devlink mutex now? IOW, why can't this series be
>>patch #1 and another patch that removes the devlink mutex?
>
>Yep, I think you are correct. We are currently working with Moshe on

Okay, I see the problem with what you suggested:
devlink_pernet_pre_exit()
There, devlink_mutex is taken to protect against simultaneous cmds
from being executed. That will be fixed with reload conversion to take
devlink->lock.


>conversion of commands that does not late devlink->lock (like health
>reporters and reload) to take devlink->lock. Once we have that, we can
>enable parallel_ops.
>
>>
>>> 
>>> 
>>> >
>>> >The devlink mutex (in accordance with the comment above it) serializes
>>> >all user space operations and accesses to the devlink devices list. This
>>> >resulted in a AA deadlock in the previous submission because we had a
>>> >flow where a user space operation (which acquires this mutex) also tries
>>> >to register / unregister a nested devlink instance which also tries to
>>> >acquire the mutex.
>>> >
>>> >As long as devlink does not implement "parallel_ops", it seems that the
>>> >devlink mutex can be reduced to only serializing accesses to the devlink
>>> >devices list, thereby eliminating the deadlock.
>>> >
>>> >> 
>>> >> The first patch makes sure that the xarray that holds devlink pointers
>>> >> is possible to be safely iterated.
>>> >> 
>>> >> The second patch moves the user command mutex to be per-devlink.
>>> >> 
>>> >> Jiri Pirko (2):
>>> >>   net: devlink: make sure that devlink_try_get() works with valid
>>> >>     pointer during xarray iteration
>>> >>   net: devlink: replace devlink_mutex by per-devlink lock
>>> >> 
>>> >>  net/core/devlink.c | 256 ++++++++++++++++++++++++++++-----------------
>>> >>  1 file changed, 161 insertions(+), 95 deletions(-)
>>> >> 
>>> >> -- 
>>> >> 2.35.3
>>> >>