[Cluster-devel] [GFS2 PATCH 6/9] gfs2: Make secondary withdrawers wait for first withdrawer

[Bob Peterson <rpeterso@redhat.com>]

Before this patch, if a process encountered an error and decided to
withdraw, if another process was already in the process of withdrawing,
the secondary withdraw would be silently ignored, which set it free
to proceed with its processing, unlock any locks, etc. That's correct
behavior if the original withdrawer encounters further errors down
the road. However, second withdrawers need to wait for the first
withdrawer to finish its withdraw before proceeding. If we don't wait
we could end up assuming everything is alright, unlock glocks and
telling other nodes they can have the glock, despite the fact that
a withdraw is still ongoing and may require a journal replay before
any locks are released. For example, if an rgrp glock is freed
by a process that didn't wait for the withdraw, a journal replay
could introduce file system corruption by replaying a rgrp block
that has already been granted to another node.

This patch makes secondary withdrawers wait until the primary
withdrawer is finished with its processing before proceeding.

Signed-off-by: Bob Peterson <rpeterso@redhat.com>
---
 fs/gfs2/incore.h | 3 +++
 fs/gfs2/util.c   | 9 ++++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h
index 2ddae1326ce2..51ca34594ac2 100644
--- a/fs/gfs2/incore.h
+++ b/fs/gfs2/incore.h
@@ -626,6 +626,7 @@ enum {
 	SDF_REMOTE_WITHDRAW	= 12, /* Performing remote recovery */
 	SDF_WITHDRAW_RECOVERY	= 13, /* Wait for journal recovery when we are
 					 withdrawing */
+	SDF_WITHDRAW_COMPLETE	= 14, /* Withdraw is compete */
 };
 
 enum gfs2_freeze_state {
@@ -836,6 +837,8 @@ struct gfs2_sbd {
 	struct bio *sd_log_bio;
 	wait_queue_head_t sd_log_flush_wait;
 	int sd_log_error;
+	atomic_t sd_withdrawer;
+	wait_queue_head_t sd_withdraw_wait;
 
 	atomic_t sd_reserving_log;
 	wait_queue_head_t sd_reserving_log_wait;
diff --git a/fs/gfs2/util.c b/fs/gfs2/util.c
index 75f67284bba8..6e0a85fc1d46 100644
--- a/fs/gfs2/util.c
+++ b/fs/gfs2/util.c
@@ -188,9 +188,15 @@ int gfs2_lm_withdraw(struct gfs2_sbd *sdp, const char *fmt, ...)
 	struct va_format vaf;
 
 	if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW &&
-	    test_and_set_bit(SDF_SHUTDOWN, &sdp->sd_flags))
+	    test_and_set_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+		fs_warn(sdp, "Waiting for process %d to withdraw.\n",
+			atomic_read(&sdp->sd_withdrawer));
+		wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_COMPLETE,
+			    TASK_UNINTERRUPTIBLE);
 		return 0;
+	}
 
+	atomic_set(&sdp->sd_withdrawer, pid_nr(task_pid(current)));
 	clear_bit(SDF_PENDING_WITHDRAW, &sdp->sd_flags);
 	if (fmt) {
 		va_start(args, fmt);
@@ -221,6 +227,7 @@ int gfs2_lm_withdraw(struct gfs2_sbd *sdp, const char *fmt, ...)
 		set_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags);
 		fs_err(sdp, "File system withdrawn\n");
 		dump_stack();
+		wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_COMPLETE);
 	}
 
 	if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
-- 
2.20.1