[Cluster-devel] gfs2 hang in xfstests generic/361

[Bob Peterson <rpeterso@redhat.com>]

On 7/26/21 9:00 AM, Christoph Hellwig wrote:
> I noticed this hang while testing the iomap_iter series on gfs2,
> but it also reproduces on 5.14-rc3.  This is running locally with
> "-O -p lock_nolock":
> 
> generic/361 files ... [ 1479.222703] run fstests generic/361 at 2021-07-26 13:57:10
(snip)
> [ 1491.752587] =============================
> [ 1491.753403] [ BUG: Invalid wait context ]
> [ 1491.754122] 5.14.0-rc2+ #47 Not tainted
> [ 1491.754860] -----------------------------
> [ 1491.755563] kworker/2:1H/1975 is trying to lock:
> [ 1491.756370] ffff8881048d0888 (&wq->mutex){+.+.}-{3:3}, at: flush_workqueue+0xc9/0x5f0

Hi Christoph,

Okay, I can recreate the failure you hit. The problem is that a withdraw
is happening from the glock workqueue, which is not right, since the
withdraw sequence itself waits for that workqueue to be flushed out.
(The withdraw itself is expected in this case, due to the IO errors.)

A patch like the following should hopefully fix the deadlock and/or BUG
by deferring the withdraw until the gfs2_logd daemon or something else
catches the error the next time it runs its loop. This is what is
supposed to happen in other similar situations we've had in the past.
Basically, we ignore the withdraw until it can be handled from a more
appropriate context.

diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c
index 42c15cfc0821..c138850b1f27 100644
--- a/fs/gfs2/log.c
+++ b/fs/gfs2/log.c
@@ -364,10 +364,8 @@ static int gfs2_ail1_empty(struct gfs2_sbd *sdp, 
int max_revokes)
  	ret = list_empty(&sdp->sd_ail1_list);
  	spin_unlock(&sdp->sd_ail_lock);

-	if (test_bit(SDF_WITHDRAWING, &sdp->sd_flags)) {
+	if (test_bit(SDF_WITHDRAWING, &sdp->sd_flags))
  		gfs2_lm(sdp, "fatal: I/O error(s)\n");
-		gfs2_withdraw(sdp);
-	}

  	return ret;
  }

Can you try something like that and see if it handles the withdraw more
sanely? Thanks.

Regards,

Bob Peterson