From: Andreas Gruenbacher <agruenba@redhat.com>
To: cluster-devel.redhat.com
Subject: [Cluster-devel] [PATCH 15/25] GFS2: Fix recovery issues for spectators
Date: Mon, 13 Aug 2018 19:45:45 +0200 [thread overview]
Message-ID: <20180813174555.10402-16-agruenba@redhat.com> (raw)
In-Reply-To: <20180813174555.10402-1-agruenba@redhat.com>
From: Bob Peterson <rpeterso@redhat.com>
This patch fixes a couple problems dealing with spectators who
remain with gfs2 mounts after the last non-spectator node fails.
Before this patch, spectator mounts would try to acquire the dlm's
mounted lock EX as part of its normal recovery sequence.
The mounted lock is only used to determine whether the node is
the first mounter, the first node to mount the file system, for
the purposes of file system recovery and journal replay.
It's not necessary for spectators: they should never do journal
recovery. If they acquire the lock it will prevent another "real"
first-mounter from acquiring the lock in EX mode, which means it
also cannot do journal recovery because it doesn't think it's the
first node to mount the file system.
This patch checks if the mounter is a spectator, and if so, avoids
grabbing the mounted lock. This allows a secondary mounter who is
really the first non-spectator mounter, to do journal recovery:
since the spectator doesn't acquire the lock, it can grab it in
EX mode, and therefore consider itself to be the first mounter
both as a "real" first mount, and as a first-real-after-spectator.
Note that the control lock still needs to be taken in PR mode
in order to fetch the lvb value so it has the current status of
all journal's recovery. This is used as it is today by a first
mounter to replay the journals. For spectators, it's merely
used to fetch the status bits. All recovery is bypassed and the
node waits until recovery is completed by a non-spectator node.
I also improved the cryptic message given by control_mount when
a spectator is waiting for a non-spectator to perform recovery.
It also fixes a problem in gfs2_recover_set whereby spectators
were never queueing recovery work for their own journal.
They cannot do recovery themselves, but they still need to queue
the work so they can check the recovery bits and clear the
DFL_BLOCK_LOCKS bit once the recovery happens on another node.
When the work queue runs on a spectator, it bypasses most of the
work so it won't print a bunch of annoying messages. All it will
print is a bunch of messages that look like this until recovery
completes on the non-spectator node:
GFS2: fsid=mycluster:scratch.s: recover generation 3 jid 0
GFS2: fsid=mycluster:scratch.s: recover jid 0 result busy
These continue every 1.5 seconds until the recovery is done by
the non-spectator, at which time it says:
GFS2: fsid=mycluster:scratch.s: recover generation 4 done
Then it proceeds with its mount.
If the file system is mounted in spectator node and the last
remaining non-spectator is fenced, any IO to the file system is
blocked by dlm and the spectator waits until recovery is
performed by a non-spectator.
If a spectator tries to mount the file system before any
non-spectators, it blocks and repeatedly gives this kernel
message:
GFS2: fsid=mycluster:scratch: Recovery is required. Waiting for a non-spectator to mount.
GFS2: fsid=mycluster:scratch: Recovery is required. Waiting for a non-spectator to mount.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
---
fs/gfs2/lock_dlm.c | 20 +++++++++++++++++---
fs/gfs2/recovery.c | 7 ++++---
fs/gfs2/sys.c | 11 +++++++++--
3 files changed, 30 insertions(+), 8 deletions(-)
diff --git a/fs/gfs2/lock_dlm.c b/fs/gfs2/lock_dlm.c
index 006c6164f759..ac7caa267ed6 100644
--- a/fs/gfs2/lock_dlm.c
+++ b/fs/gfs2/lock_dlm.c
@@ -821,6 +821,13 @@ static int control_mount(struct gfs2_sbd *sdp)
goto fail;
}
+ /**
+ * If we're a spectator, we don't want to take the lock in EX because
+ * we cannot do the first-mount responsibility it implies: recovery.
+ */
+ if (sdp->sd_args.ar_spectator)
+ goto locks_done;
+
error = mounted_lock(sdp, DLM_LOCK_EX, DLM_LKF_CONVERT|DLM_LKF_NOQUEUE);
if (!error) {
mounted_mode = DLM_LOCK_EX;
@@ -896,9 +903,16 @@ static int control_mount(struct gfs2_sbd *sdp)
if (lvb_gen < mount_gen) {
/* wait for mounted nodes to update control_lock lvb to our
generation, which might include new recovery bits set */
- fs_info(sdp, "control_mount wait1 block %u start %u mount %u "
- "lvb %u flags %lx\n", block_gen, start_gen, mount_gen,
- lvb_gen, ls->ls_recover_flags);
+ if (sdp->sd_args.ar_spectator) {
+ fs_info(sdp, "Recovery is required. Waiting for a "
+ "non-spectator to mount.\n");
+ msleep_interruptible(1000);
+ } else {
+ fs_info(sdp, "control_mount wait1 block %u start %u "
+ "mount %u lvb %u flags %lx\n", block_gen,
+ start_gen, mount_gen, lvb_gen,
+ ls->ls_recover_flags);
+ }
spin_unlock(&ls->ls_recover_spin);
goto restart;
}
diff --git a/fs/gfs2/recovery.c b/fs/gfs2/recovery.c
index d8b622c375ab..0f501f938d1c 100644
--- a/fs/gfs2/recovery.c
+++ b/fs/gfs2/recovery.c
@@ -413,12 +413,13 @@ void gfs2_recover_func(struct work_struct *work)
ktime_t t_start, t_jlck, t_jhd, t_tlck, t_rep;
int ro = 0;
unsigned int pass;
- int error;
+ int error = 0;
int jlocked = 0;
t_start = ktime_get();
- if (sdp->sd_args.ar_spectator ||
- (jd->jd_jid != sdp->sd_lockstruct.ls_jid)) {
+ if (sdp->sd_args.ar_spectator)
+ goto fail;
+ if (jd->jd_jid != sdp->sd_lockstruct.ls_jid) {
fs_info(sdp, "jid=%u: Trying to acquire journal lock...\n",
jd->jd_jid);
jlocked = 1;
diff --git a/fs/gfs2/sys.c b/fs/gfs2/sys.c
index c191fa58a1df..1787d295834e 100644
--- a/fs/gfs2/sys.c
+++ b/fs/gfs2/sys.c
@@ -429,11 +429,18 @@ int gfs2_recover_set(struct gfs2_sbd *sdp, unsigned jid)
spin_lock(&sdp->sd_jindex_spin);
rv = -EBUSY;
- if (sdp->sd_jdesc->jd_jid == jid)
+ /**
+ * If we're a spectator, we use journal0, but it's not really ours.
+ * So we need to wait for its recovery too. If we skip it we'd never
+ * queue work to the recovery workqueue, and so its completion would
+ * never clear the DFL_BLOCK_LOCKS flag, so all our locks would
+ * permanently stop working.
+ */
+ if (sdp->sd_jdesc->jd_jid == jid && !sdp->sd_args.ar_spectator)
goto out;
rv = -ENOENT;
list_for_each_entry(jd, &sdp->sd_jindex_list, jd_list) {
- if (jd->jd_jid != jid)
+ if (jd->jd_jid != jid && !sdp->sd_args.ar_spectator)
continue;
rv = gfs2_recover_journal(jd, false);
break;
--
2.17.1
next prev parent reply other threads:[~2018-08-13 17:45 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-13 17:45 [Cluster-devel] [PATCH 00/25] GFS2: Pre-pull patch posting (merge window) Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 01/25] gfs2: eliminate rs_inum and reduce the size of gfs2 inodes Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 02/25] gfs2: Don't withdraw under a spin lock Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 03/25] gfs2: Minor clarification to __gfs2_punch_hole Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 04/25] gfs2: call ktime_get_coarse_real_ts64() directly Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 05/25] gfs2: Further iomap cleanups Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 06/25] gfs2: iomap buffered write support Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 07/25] gfs2: gfs2_extent_length cleanup Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 08/25] gfs2: iomap direct I/O support Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 09/25] gfs2: Remove gfs2_write_{begin,end} Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 10/25] gfs2: Stop messing with ip->i_rgd in the rlist code Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 11/25] gfs2: Eliminate redundant ip->i_rgd Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 12/25] gfs2: Don't reject a supposedly full bitmap if we have blocks reserved Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 13/25] gfs2: using posix_acl_xattr_size instead of posix_acl_to_xattr Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 14/25] fs: gfs2: Adding new return type vm_fault_t Andreas Gruenbacher
2018-08-13 17:45 ` Andreas Gruenbacher [this message]
2018-08-13 17:45 ` [Cluster-devel] [PATCH 16/25] gfs2: fallocate_chunk: Always initialize struct iomap Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 17/25] gfs2: Use iomap for stuffed direct I/O reads Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 18/25] gfs2: use iomap_readpage for blocksize == PAGE_SIZE Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 19/25] gfs2: remove redundant variable 'moved' Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 20/25] GFS2: rgrp free blocks used incorrectly Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 21/25] gfs2: Special-case rindex for gfs2_grow Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 22/25] gfs2: cleanup: call gfs2_rgrp_ondisk2lvb from gfs2_rgrp_out Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 23/25] gfs2: Get rid of gfs2_ea_strlen Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 24/25] gfs2: Fix gfs2_testbit to use clone bitmaps Andreas Gruenbacher
2018-08-13 17:45 ` [Cluster-devel] [PATCH 25/25] gfs2: eliminate update_rgrp_lvb_unlinked Andreas Gruenbacher
2018-08-16 13:53 ` [Cluster-devel] [PATCH 00/25] GFS2: Pre-pull patch posting (merge window) Andreas Gruenbacher
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180813174555.10402-16-agruenba@redhat.com \
--to=agruenba@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).