From: "Darrick J. Wong" <djwong@kernel.org>
To: Dave Chinner <david@fromorbit.com>,
Allison Henderson <allison.henderson@oracle.com>
Cc: linux-xfs@vger.kernel.org
Subject: Re: xfs: fix CIL push hang in for-next tree
Date: Tue, 15 Jun 2021 10:57:19 -0700 [thread overview]
Message-ID: <20210615175719.GD158209@locust> (raw)
In-Reply-To: <20210615064658.854029-1-david@fromorbit.com>
On Tue, Jun 15, 2021 at 04:46:56PM +1000, Dave Chinner wrote:
> Hi folks,
>
> This is the first fix for the problems Brian has reported from
> generic/019. This has fixed the hang, but the other log recovery
> problem he reported is still present (seen once with these patches
> in place).
>
> I've tested these out to a couple of hundred cycles of
> continual looping generic/019 before the systems fall over with a
> perag reference count underrun at unmount after a shutdown. I'm
> pretty sure the hang is fixed, as it would manifest within 10-20
> cycles without this patch.
>
> The first patch is the iclogbuf state tracing I used to capture the
> iclogbuf wrapping state. The second patch is the fix.
I found another bug while testing for-next. If I run generic/100 more
than about ~30 times with a 1k block size:
FSTYP -- xfs (debug)
PLATFORM -- Linux/x86_64 flax-mtr00 5.13.0-rc4-djwx #rc4 SMP
PREEMPT Mon Jun 7 11:17:23 PDT 2021
MKFS_OPTIONS -- -f -b size=1024, /dev/sdf
MOUNT_OPTIONS -- -o usrquota,grpquota,prjquota, /dev/sdf /opt
I see this in dmesg:
run fstests generic/100 at 2021-06-15 10:41:45
XFS (sda): ctx ticket reservation ran out. Need to up reservation
XFS (sda): ticket reservation summary:
XFS (sda): unit res = 47168 bytes
XFS (sda): current res = -404 bytes
XFS (sda): original count = 1
XFS (sda): remaining count = 1
XFS (sda): xfs_do_force_shutdown(0x2) called from line 2440 of file fs/xfs/xfs_log.c. Return address = xlog_write+0x608/0x640 [xfs]
XFS (sda): Log I/O Error Detected. Shutting down filesystem
XFS (sda): Please unmount the filesystem and rectify the problem(s)
XFS (sda): Unmounting Filesystem
Looking up that line in gdb produces:
0xffffffffa038a0a8 is in xlog_write (fs/xfs/xfs_log.c:2439).
2434 int log_offset;
2435
2436 if (ticket->t_curr_res < 0) {
2437 xfs_alert_tag(log->l_mp, XFS_PTAG_LOGRES,
2438 "ctx ticket reservation ran out. Need to up reservation");
2439 xlog_print_tic_res(log->l_mp, ticket);
2440 xfs_force_shutdown(log->l_mp, SHUTDOWN_LOG_IO_ERROR);
2441 }
I haven't applied these two patches yet, but looking back through
fstests reports I never saw this before the recent for-next push.
I'm uncertain if it's the CIL work or the xattr refactoring that did
this, though AFAICT generic/100 itself does not generate any xattrs and
I don't have any LSMs enabled that would cause them to be created.
--D
>
> Cheers,
>
> Dave.
>
>
prev parent reply other threads:[~2021-06-15 17:57 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-15 6:46 xfs: fix CIL push hang in for-next tree Dave Chinner
2021-06-15 6:46 ` [PATCH 1/2] xfs: add iclog state trace events Dave Chinner
2021-06-15 15:26 ` Darrick J. Wong
2021-06-15 15:37 ` Brian Foster
2021-06-15 22:47 ` Dave Chinner
2021-06-16 14:55 ` Brian Foster
2021-06-15 6:46 ` [PATCH 2/2] xfs: don't wait on future iclogs when pushing the CIL Dave Chinner
2021-06-15 15:38 ` Brian Foster
2021-06-15 22:09 ` Dave Chinner
2021-06-16 14:57 ` Brian Foster
2021-06-16 22:20 ` Dave Chinner
2021-06-17 13:15 ` Brian Foster
2021-06-15 16:19 ` Darrick J. Wong
2021-06-15 17:57 ` Darrick J. Wong [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210615175719.GD158209@locust \
--to=djwong@kernel.org \
--cc=allison.henderson@oracle.com \
--cc=david@fromorbit.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.