public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Dave Chinner <dchinner@redhat.com>,
	"Darrick J. Wong" <djwong@kernel.org>,
	Amir Goldstein <amir73il@gmail.com>
Subject: [PATCH 5.10 10/12] xfs: check sb_meta_uuid for dabuf buffer recovery
Date: Thu, 30 Jun 2022 15:47:15 +0200	[thread overview]
Message-ID: <20220630133230.996324972@linuxfoundation.org> (raw)
In-Reply-To: <20220630133230.676254336@linuxfoundation.org>

From: Dave Chinner <dchinner@redhat.com>

commit 09654ed8a18cfd45027a67d6cbca45c9ea54feab upstream.

Got a report that a repeated crash test of a container host would
eventually fail with a log recovery error preventing the system from
mounting the root filesystem. It manifested as a directory leaf node
corruption on writeback like so:

 XFS (loop0): Mounting V5 Filesystem
 XFS (loop0): Starting recovery (logdev: internal)
 XFS (loop0): Metadata corruption detected at xfs_dir3_leaf_check_int+0x99/0xf0, xfs_dir3_leaf1 block 0x12faa158
 XFS (loop0): Unmount and run xfs_repair
 XFS (loop0): First 128 bytes of corrupted metadata buffer:
 00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b  ........=.......
 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc  .......X...)....
 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23  ..x..~J}.S...G.#
 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00  .........C......
 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a  ................
 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50  .5y....0.......P
 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4  .@.......A......
 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c  .b.......P!A....
 XFS (loop0): Corruption of in-memory data (0x8) detected at xfs_do_force_shutdown+0x1a/0x20 (fs/xfs/xfs_buf.c:1514).  Shutting down.
 XFS (loop0): Please unmount the filesystem and rectify the problem(s)
 XFS (loop0): log mount/recovery failed: error -117
 XFS (loop0): log mount failed

Tracing indicated that we were recovering changes from a transaction
at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57.
That is, log recovery was overwriting a buffer with newer changes on
disk than was in the transaction. Tracing indicated that we were
hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the
buffer.

The code was extracting the LSN correctly, then ignoring it because
the UUID in the buffer did not match the superblock UUID. The
problem arises because the UUID check uses the wrong UUID - it
should be checking the sb_meta_uuid, not sb_uuid. This filesystem
has sb_uuid != sb_meta_uuid (which is fine), and the buffer has the
correct matching sb_meta_uuid in it, it's just the code checked it
against the wrong superblock uuid.

The is no corruption in the filesystem, and failing to recover the
buffer due to a write verifier failure means the recovery bug did
not propagate the corruption to disk. Hence there is no corruption
before or after this bug has manifested, the impact is limited
simply to an unmountable filesystem....

This was missed back in 2015 during an audit of incorrect sb_uuid
usage that resulted in commit fcfbe2c4ef42 ("xfs: log recovery needs
to validate against sb_meta_uuid") that fixed the magic32 buffers to
validate against sb_meta_uuid instead of sb_uuid. It missed the
magicda buffers....

Fixes: ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Acked-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 fs/xfs/xfs_buf_item_recover.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/fs/xfs/xfs_buf_item_recover.c
+++ b/fs/xfs/xfs_buf_item_recover.c
@@ -805,7 +805,7 @@ xlog_recover_get_buf_lsn(
 	}
 
 	if (lsn != (xfs_lsn_t)-1) {
-		if (!uuid_equal(&mp->m_sb.sb_uuid, uuid))
+		if (!uuid_equal(&mp->m_sb.sb_meta_uuid, uuid))
 			goto recover_immediately;
 		return lsn;
 	}



  parent reply	other threads:[~2022-06-30 14:06 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-30 13:47 [PATCH 5.10 00/12] 5.10.128-rc1 review Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 01/12] MAINTAINERS: add Amir as xfs maintainer for 5.10.y Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 02/12] drm: remove drm_fb_helper_modinit Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 03/12] tick/nohz: unexport __init-annotated tick_nohz_full_setup() Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 04/12] clocksource/drivers/ixp4xx: remove __init from ixp4xx_timer_setup() Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 05/12] bcache: memset on stack variables in bch_btree_check() and bch_sectors_dirty_init() Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 06/12] xfs: use kmem_cache_free() for kmem_cache objects Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 07/12] xfs: punch out data fork delalloc blocks on COW writeback failure Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 08/12] xfs: Fix the free logic of state in xfs_attr_node_hasname Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 09/12] xfs: remove all COW fork extents when remounting readonly Greg Kroah-Hartman
2022-06-30 13:47 ` Greg Kroah-Hartman [this message]
2022-06-30 13:47 ` [PATCH 5.10 11/12] powerpc/ftrace: Remove ftrace init tramp once kernel init is complete Greg Kroah-Hartman
2022-06-30 13:47 ` [PATCH 5.10 12/12] net: mscc: ocelot: allow unregistered IP multicast flooding Greg Kroah-Hartman
2022-06-30 23:03 ` [PATCH 5.10 00/12] 5.10.128-rc1 review Florian Fainelli
2022-06-30 23:15 ` Shuah Khan
2022-07-01  0:57 ` Guenter Roeck
2022-07-01  3:16 ` Samuel Zou
2022-07-01  6:23 ` Naresh Kamboju
2022-07-01 10:34 ` Sudip Mukherjee
2022-07-01 18:47 ` Pavel Machek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220630133230.996324972@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=amir73il@gmail.com \
    --cc=dchinner@redhat.com \
    --cc=djwong@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox