[PATCH dlm/next] dlm: fix missing rsb put on scan timer

[PATCH dlm/next] dlm: fix missing rsb put on scan timer

[Alexander Aring]

I figured out we don't cleanup all rsb on toss list right now if all
locks are unlocked in the cluster for a resource. After investigation
I figured out the condition to add_scan() is wrong and I added a comment
for the second important condition for all possible permuations of the
boolean expression.

The current expression:

"(r->res_master_nodeid != our_nodeid &&
  dlm_dir_nodeid(r) != our_nodeid))"

it only covers the case when we are not the master and not the dir node
which is the second case of the added comment in 2. Case 1 and 3 are
forgotten.

Remove also the del_scan() as it always should remove the rsb out of the
scan timer when it's on, if it's not on del_scan() will do nothing.

For find_rsb_nodir() the del_scan() need to be moved before clearing the
RSB_INACTIVE flag as del_scan() is has a WARN_ON() on it.

Fixes: c217adfc8caa ("dlm: fix add_scan and del_scan usage")
Signed-off-by: Alexander Aring <aahringo@redhat.com>
---
 fs/dlm/lock.c | 53 ++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 38 insertions(+), 15 deletions(-)

diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index b406322f26a7..5ea3919b62a0 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -906,9 +906,12 @@ static int find_rsb_dir(struct dlm_ls *ls, const void *name, int len,
 		r->res_first_lkid = 0;
 	}
 
-	/* A dir record will not be on the scan list. */
-	if (r->res_dir_nodeid != our_nodeid)
-		del_scan(ls, r);
+	/* we always deactivate scan timer for the rsb, when
+	 * we move it out of the inactive state as rsb state
+	 * can be changed and scan timers are only for inactive
+	 * rsbs.
+	 */
+	del_scan(ls, r);
 	list_move(&r->res_slow_list, &ls->ls_slow_active);
 	rsb_clear_flag(r, RSB_INACTIVE);
 	kref_init(&r->res_ref); /* ref is now used in active state */
@@ -1071,10 +1074,10 @@ static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len,
 		r->res_nodeid = 0;
 	}
 
+	del_scan(ls, r);
 	list_move(&r->res_slow_list, &ls->ls_slow_active);
 	rsb_clear_flag(r, RSB_INACTIVE);
 	kref_init(&r->res_ref);
-	del_scan(ls, r);
 	write_unlock_bh(&ls->ls_rsbtbl_lock);
 
 	goto out;
@@ -1419,9 +1422,13 @@ static int _dlm_master_lookup(struct dlm_ls *ls, int from_nodeid, const char *na
 	__dlm_master_lookup(ls, r, our_nodeid, from_nodeid, true, flags,
 			    r_nodeid, result);
 
-	/* A dir record rsb should never be on scan list. */
-	/* Try to fix this with del_scan? */
-	WARN_ON(!list_empty(&r->res_scan_list));
+	/* A dir record rsb should never be on scan list.
+	 * Except when we are the dir and master node.
+	 * This function should only be called by the dir
+	 * node.
+	 */
+	WARN_ON(!list_empty(&r->res_scan_list) &&
+		r->res_master_nodeid != our_nodeid);
 
 	write_unlock_bh(&ls->ls_rsbtbl_lock);
 
@@ -1513,15 +1520,31 @@ static void deactivate_rsb(struct kref *kref)
 
 	/*
 	 * When the rsb becomes unused:
-	 * - If it's not a dir record for a remote master rsb,
-	 *   then it is put on the scan list to be freed.
-	 * - If it's a dir record for a remote master rsb,
-	 *   then it is kept in the inactive state until
-	 *   receive_remove() from the master node.
+	 * - It's on scan list when there is no directory
+	 *   node functionality
+	 * or
+	 * - If we are the rsb master we need to call
+	 *   send_remove() to the dir node and delete ourself.
+	 *
+	 *   The second case is if we are not the dir node we
+	 *   need to delete ourself as well but don't call
+	 *   send_remove() as we are not the master.
+	 *
+	 *   There is a thrid case when the node is master and
+	 *   dir node at the same time. Then the rsb need to be
+	 *   added to the scan timer as well because the scan
+	 *   acts like a local send_remove() that is done
+	 *   immediately.
+	 *
+	 *   The fourth case is the one we need to avoid here,
+	 *   when we are the directory node and not the master.
+	 *   Then we never should be on the scan timer and
+	 *   waiting of receive_remove() of the second case.
+	 *   Then the condition is false.
 	 */
-	if (!dlm_no_directory(ls) &&
-	    (r->res_master_nodeid != our_nodeid) &&
-	    (dlm_dir_nodeid(r) != our_nodeid))
+	if (dlm_no_directory(ls) ||
+	    (r->res_master_nodeid == our_nodeid ||
+	     dlm_dir_nodeid(r) != our_nodeid))
 		add_scan(ls, r);
 
 	if (r->res_lvbptr) {
-- 
2.43.0

Re: [PATCH dlm/next] dlm: fix missing rsb put on scan timer

[Alexander Aring]

Hi,

On Tue, Nov 19, 2024 at 3:56 PM Alexander Aring <aahringo@redhat.com> wrote:
>
> I figured out we don't cleanup all rsb on toss list right now if all
> locks are unlocked in the cluster for a resource. After investigation
> I figured out the condition to add_scan() is wrong and I added a comment
> for the second important condition for all possible permuations of the
> boolean expression.
>
> The current expression:
>
> "(r->res_master_nodeid != our_nodeid &&
>   dlm_dir_nodeid(r) != our_nodeid))"
>
> it only covers the case when we are not the master and not the dir node
> which is the second case of the added comment in 2. Case 1 and 3 are
> forgotten.
>
> Remove also the del_scan() as it always should remove the rsb out of the
> scan timer when it's on, if it's not on del_scan() will do nothing.
>
> For find_rsb_nodir() the del_scan() need to be moved before clearing the
> RSB_INACTIVE flag as del_scan() is has a WARN_ON() on it.
>
> Fixes: c217adfc8caa ("dlm: fix add_scan and del_scan usage")
> Signed-off-by: Alexander Aring <aahringo@redhat.com>
> ---
>  fs/dlm/lock.c | 53 ++++++++++++++++++++++++++++++++++++---------------
>  1 file changed, 38 insertions(+), 15 deletions(-)
>
> diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
> index b406322f26a7..5ea3919b62a0 100644
> --- a/fs/dlm/lock.c
> +++ b/fs/dlm/lock.c
> @@ -906,9 +906,12 @@ static int find_rsb_dir(struct dlm_ls *ls, const void *name, int len,
>                 r->res_first_lkid = 0;
>         }
>
> -       /* A dir record will not be on the scan list. */
> -       if (r->res_dir_nodeid != our_nodeid)
> -               del_scan(ls, r);
> +       /* we always deactivate scan timer for the rsb, when
> +        * we move it out of the inactive state as rsb state
> +        * can be changed and scan timers are only for inactive
> +        * rsbs.
> +        */
> +       del_scan(ls, r);
>         list_move(&r->res_slow_list, &ls->ls_slow_active);
>         rsb_clear_flag(r, RSB_INACTIVE);
>         kref_init(&r->res_ref); /* ref is now used in active state */
> @@ -1071,10 +1074,10 @@ static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len,
>                 r->res_nodeid = 0;
>         }
>
> +       del_scan(ls, r);
>         list_move(&r->res_slow_list, &ls->ls_slow_active);
>         rsb_clear_flag(r, RSB_INACTIVE);
>         kref_init(&r->res_ref);
> -       del_scan(ls, r);
>         write_unlock_bh(&ls->ls_rsbtbl_lock);
>
>         goto out;
> @@ -1419,9 +1422,13 @@ static int _dlm_master_lookup(struct dlm_ls *ls, int from_nodeid, const char *na
>         __dlm_master_lookup(ls, r, our_nodeid, from_nodeid, true, flags,
>                             r_nodeid, result);
>
> -       /* A dir record rsb should never be on scan list. */
> -       /* Try to fix this with del_scan? */
> -       WARN_ON(!list_empty(&r->res_scan_list));
> +       /* A dir record rsb should never be on scan list.
> +        * Except when we are the dir and master node.
> +        * This function should only be called by the dir
> +        * node.
> +        */
> +       WARN_ON(!list_empty(&r->res_scan_list) &&
> +               r->res_master_nodeid != our_nodeid);
>
>         write_unlock_bh(&ls->ls_rsbtbl_lock);
>
> @@ -1513,15 +1520,31 @@ static void deactivate_rsb(struct kref *kref)
>
>         /*
>          * When the rsb becomes unused:
> -        * - If it's not a dir record for a remote master rsb,
> -        *   then it is put on the scan list to be freed.
> -        * - If it's a dir record for a remote master rsb,
> -        *   then it is kept in the inactive state until
> -        *   receive_remove() from the master node.
> +        * - It's on scan list when there is no directory
> +        *   node functionality
> +        * or
> +        * - If we are the rsb master we need to call
> +        *   send_remove() to the dir node and delete ourself.
> +        *
> +        *   The second case is if we are not the dir node we
> +        *   need to delete ourself as well but don't call
> +        *   send_remove() as we are not the master.
> +        *
> +        *   There is a thrid case when the node is master and
> +        *   dir node at the same time. Then the rsb need to be
> +        *   added to the scan timer as well because the scan
> +        *   acts like a local send_remove() that is done
> +        *   immediately.
> +        *
> +        *   The fourth case is the one we need to avoid here,
> +        *   when we are the directory node and not the master.
> +        *   Then we never should be on the scan timer and
> +        *   waiting of receive_remove() of the second case.

meant the first case here...

> +        *   Then the condition is false.
>          */
> -       if (!dlm_no_directory(ls) &&
> -           (r->res_master_nodeid != our_nodeid) &&
> -           (dlm_dir_nodeid(r) != our_nodeid))
> +       if (dlm_no_directory(ls) ||
> +           (r->res_master_nodeid == our_nodeid ||
> +            dlm_dir_nodeid(r) != our_nodeid))
>                 add_scan(ls, r);

- Alex