[Cluster-devel] [PATCH 06/15] gfs2: Make secondary withdrawers wait for first withdrawer

[Bob Peterson <rpeterso@redhat.com>]

Before this patch, if a process encountered an error and decided to
withdraw, if another process was already in the process of withdrawing,
the secondary withdraw would be silently ignored, which set it free
to proceed with its processing, unlock any locks, etc. That's correct
behavior if the original withdrawer encounters further errors down
the road. However, second withdrawers need to wait for the first
withdrawer to finish its withdraw before proceeding. If we don't wait
we could end up assuming everything is alright, unlock glocks and
telling other nodes they can have the glock, despite the fact that
a withdraw is still ongoing and may require a journal replay before
any locks are released. For example, if an rgrp glock is freed
by a process that didn't wait for the withdraw, a journal replay
could introduce file system corruption by replaying a rgrp block
that has already been granted to another node.

This patch makes secondary withdrawers wait until the primary
withdrawer is finished with its processing before proceeding.

Signed-off-by: Bob Peterson <rpeterso@redhat.com>
---
 fs/gfs2/incore.h | 3 +++
 fs/gfs2/util.c   | 9 ++++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h
index d84430eceb6c..ef0a88fcd0ca 100644
--- a/fs/gfs2/incore.h
+++ b/fs/gfs2/incore.h
@@ -622,6 +622,7 @@ enum {
 	SDF_SKIP_DLM_UNLOCK	= 8,
 	SDF_FORCE_AIL_FLUSH     = 9,
 	SDF_PENDING_WITHDRAW	= 10, /* Will withdraw eventually */
+	SDF_WITHDRAW_COMPLETE	= 11, /* Withdraw is complete */
 };
 
 enum gfs2_freeze_state {
@@ -832,6 +833,8 @@ struct gfs2_sbd {
 	wait_queue_head_t sd_log_flush_wait;
 	int sd_log_error; /* First log error */
 	atomic_t sd_log_errors; /* Count of log errors */
+	atomic_t sd_withdrawer;
+	wait_queue_head_t sd_withdraw_wait;
 
 	atomic_t sd_reserving_log;
 	wait_queue_head_t sd_reserving_log_wait;
diff --git a/fs/gfs2/util.c b/fs/gfs2/util.c
index dc00747d35f3..eb92c85ad9bb 100644
--- a/fs/gfs2/util.c
+++ b/fs/gfs2/util.c
@@ -89,9 +89,15 @@ int gfs2_lm_withdraw(struct gfs2_sbd *sdp, const char *fmt, ...)
 	struct va_format vaf;
 
 	if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW &&
-	    test_and_set_bit(SDF_SHUTDOWN, &sdp->sd_flags))
+	    test_and_set_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+		fs_warn(sdp, "Waiting for process %d to withdraw.\n",
+			atomic_read(&sdp->sd_withdrawer));
+		wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_COMPLETE,
+			    TASK_UNINTERRUPTIBLE);
 		return 0;
+	}
 
+	atomic_set(&sdp->sd_withdrawer, pid_nr(task_pid(current)));
 	clear_bit(SDF_PENDING_WITHDRAW, &sdp->sd_flags);
 	if (fmt) {
 		va_start(args, fmt);
@@ -120,6 +126,7 @@ int gfs2_lm_withdraw(struct gfs2_sbd *sdp, const char *fmt, ...)
 		set_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags);
 		fs_err(sdp, "withdrawn\n");
 		dump_stack();
+		wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_COMPLETE);
 	}
 
 	if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
-- 
2.20.1