* [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features
@ 2023-08-29 23:09 Darrick J. Wong
2023-08-29 23:09 ` [PATCH 1/2] xfs/270: actually test file readability Darrick J. Wong
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Darrick J. Wong @ 2023-08-29 23:09 UTC (permalink / raw)
To: zlang, djwong; +Cc: Dave Chinner, linux-xfs, fstests, guan, david, sandeen
Hi all,
Dave pointed out some failures in xfs/270 when he upgraded Debian
unstable and util-linux started using the new mount apis. Upon further
inquiry I noticed that XFS is quite a hot mess when it encounters a
filesystem with unrecognized rocompat bits set in the superblock.
Whereas we used to allow readonly mounts under these conditions, a
change to the sb write verifier several years ago resulted in the
filesystem going down immediately because the post-mount log cleaning
writes the superblock, which trips the sb write verifier on the
unrecognized rocompat bit. I made the observation that the ROCOMPAT
features RMAPBT and REFLINK both protect new log intent item types,
which means that we actually cannot support recovering the log if we
don't recognize all the rocompat bits.
Therefore -- fix inode inactivation to work when we're recovering the
log, disallow recovery when there's unrecognized rocompat bits, and
don't clean the log if doing so would trip the rocompat checks.
v2: change direction of series to allow log recovery on ro mounts
If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy!
Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts
---
tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++---------------
tests/xfs/270.out | 2 +
2 files changed, 59 insertions(+), 23 deletions(-)
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 1/2] xfs/270: actually test file readability 2023-08-29 23:09 [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features Darrick J. Wong @ 2023-08-29 23:09 ` Darrick J. Wong 2023-09-13 16:54 ` Zorro Lang 2023-08-29 23:09 ` [PATCH 2/2] xfs/270: actually test log recovery with unknown rocompat features Darrick J. Wong 2023-09-17 16:44 ` [PATCHSET v2 0/2] fstests: fix ro mounting " Zorro Lang 2 siblings, 1 reply; 10+ messages in thread From: Darrick J. Wong @ 2023-08-29 23:09 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan, david, sandeen From: Darrick J. Wong <djwong@kernel.org> Make sure we can actually read files off the ro mounted filesystem that has an unknown rocompat feature set. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/270 | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/xfs/270 b/tests/xfs/270 index 7447ce87be..511dfe9fcd 100755 --- a/tests/xfs/270 +++ b/tests/xfs/270 @@ -23,6 +23,9 @@ _require_scratch_nocheck _require_scratch_xfs_crc _scratch_mkfs_xfs >>$seqres.full 2>&1 +_scratch_mount +echo moo > $SCRATCH_MNT/testfile +_scratch_unmount # set the highest bit of features_ro_compat, use it as an unknown # feature bit. If one day this bit become known feature, please @@ -68,6 +71,7 @@ if [ $? -ne 0 ]; then _fail "ro mount test failed" else # no hang/panic is fine + cat $SCRATCH_MNT/testfile > /dev/null $FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 400 >>$seqres.full 2>&1 fi ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 1/2] xfs/270: actually test file readability 2023-08-29 23:09 ` [PATCH 1/2] xfs/270: actually test file readability Darrick J. Wong @ 2023-09-13 16:54 ` Zorro Lang 0 siblings, 0 replies; 10+ messages in thread From: Zorro Lang @ 2023-09-13 16:54 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Tue, Aug 29, 2023 at 04:09:53PM -0700, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Make sure we can actually read files off the ro mounted filesystem that > has an unknown rocompat feature set. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Looks good to me, Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/270 | 4 ++++ > 1 file changed, 4 insertions(+) > > > diff --git a/tests/xfs/270 b/tests/xfs/270 > index 7447ce87be..511dfe9fcd 100755 > --- a/tests/xfs/270 > +++ b/tests/xfs/270 > @@ -23,6 +23,9 @@ _require_scratch_nocheck > _require_scratch_xfs_crc > > _scratch_mkfs_xfs >>$seqres.full 2>&1 > +_scratch_mount > +echo moo > $SCRATCH_MNT/testfile > +_scratch_unmount > > # set the highest bit of features_ro_compat, use it as an unknown > # feature bit. If one day this bit become known feature, please > @@ -68,6 +71,7 @@ if [ $? -ne 0 ]; then > _fail "ro mount test failed" > else > # no hang/panic is fine > + cat $SCRATCH_MNT/testfile > /dev/null > $FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 400 >>$seqres.full 2>&1 > fi > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 2/2] xfs/270: actually test log recovery with unknown rocompat features 2023-08-29 23:09 [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features Darrick J. Wong 2023-08-29 23:09 ` [PATCH 1/2] xfs/270: actually test file readability Darrick J. Wong @ 2023-08-29 23:09 ` Darrick J. Wong 2023-09-17 16:44 ` [PATCHSET v2 0/2] fstests: fix ro mounting " Zorro Lang 2 siblings, 0 replies; 10+ messages in thread From: Darrick J. Wong @ 2023-08-29 23:09 UTC (permalink / raw) To: zlang, djwong; +Cc: Dave Chinner, linux-xfs, fstests, guan, david, sandeen From: Darrick J. Wong <djwong@kernel.org> Make sure that log recovery will not succeed if there are unknown rocompat features in the superblock and the log is dirty. Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> --- tests/xfs/270 | 82 ++++++++++++++++++++++++++++++++++++----------------- tests/xfs/270.out | 2 + 2 files changed, 58 insertions(+), 26 deletions(-) diff --git a/tests/xfs/270 b/tests/xfs/270 index 511dfe9fcd..08bfb1c4d4 100755 --- a/tests/xfs/270 +++ b/tests/xfs/270 @@ -21,41 +21,48 @@ _supported_fs xfs _require_scratch_nocheck # Only V5 XFS disallow rw mount/remount with unknown ro-compat features _require_scratch_xfs_crc - -_scratch_mkfs_xfs >>$seqres.full 2>&1 -_scratch_mount -echo moo > $SCRATCH_MNT/testfile -_scratch_unmount +_require_scratch_shutdown # set the highest bit of features_ro_compat, use it as an unknown # feature bit. If one day this bit become known feature, please # change this case. +set_bad_rocompat() { + ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0") + echo $ro_compat | grep -q -E '^0x[[:xdigit:]]$' + if [[ $? != 0 ]]; then + echo "features_ro_compat has an invalid value." + return 1 + fi -ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0") -echo $ro_compat | grep -q -E '^0x[[:xdigit:]]$' -if [[ $? != 0 ]]; then - echo "features_ro_compat has an invalid value." -fi + ro_compat=$(echo $ro_compat | \ + awk '/^0x[[:xdigit:]]+/ { + printf("0x%x\n", or(strtonum($1), 0x80000000)) + }') -ro_compat=$(echo $ro_compat | \ - awk '/^0x[[:xdigit:]]+/ { - printf("0x%x\n", or(strtonum($1), 0x80000000)) - }') + # write the new ro compat field to the superblock + _scratch_xfs_set_metadata_field "features_ro_compat" "$ro_compat" "sb 0" \ + > $seqres.full 2>&1 -# write the new ro compat field to the superblock -_scratch_xfs_set_metadata_field "features_ro_compat" "$ro_compat" "sb 0" \ - > $seqres.full 2>&1 + # read the newly set ro compat filed for verification + new_ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0" \ + 2>/dev/null) -# read the newly set ro compat filed for verification -new_ro_compat=$(_scratch_xfs_get_metadata_field "features_ro_compat" "sb 0" \ - 2>/dev/null) + # verify the new ro_compat field is correct. Without xfsprogs commit + # f4afdcb0ad ("xfs_db: clean up the salvage read callsites in set_cur()"), + # we can't get new_ro_compat value. + if [ "$new_ro_compat" != "$ro_compat" ]; then + echo "Unable to set new features_ro_compat. Wanted $ro_compat, got $new_ro_compat" + return 1 + fi + return 0 +} -# verify the new ro_compat field is correct. Without xfsprogs commit -# f4afdcb0ad ("xfs_db: clean up the salvage read callsites in set_cur()"), -# we can't get new_ro_compat value. -if [ "$new_ro_compat" != "$ro_compat" ]; then - echo "Unable to set new features_ro_compat. Wanted $ro_compat, got $new_ro_compat" -fi +# Once with a clean filesystem... +_scratch_mkfs_xfs >>$seqres.full 2>&1 +_scratch_mount +echo moo > $SCRATCH_MNT/testfile +_scratch_unmount +set_bad_rocompat # rw mount with unknown ro-compat feature should fail echo "rw mount test" @@ -85,6 +92,29 @@ fi _scratch_unmount +# And again with a dirty filesystem... +_scratch_mkfs_xfs >>$seqres.full 2>&1 +_scratch_mount +echo moo > $SCRATCH_MNT/testfile +$XFS_IO_PROG -x -c 'shutdown -f' "${SCRATCH_MNT}" +_scratch_unmount +set_bad_rocompat + +# rw mount with unknown ro-compat feature should fail +echo "rw mount test" +_try_scratch_mount 2>>$seqres.full +if [ $? -eq 0 ]; then + _fail "rw mount test failed" +fi + +# ro mount should succeed even with log recovery +echo "ro mount test" +_try_scratch_mount -o ro 2>>$seqres.full +if [ $? -ne 0 ]; then + _fail "ro mount test failed" +fi +cat $SCRATCH_MNT/testfile > /dev/null + # success, all done status=0 exit diff --git a/tests/xfs/270.out b/tests/xfs/270.out index edf4c25489..a519d2f328 100644 --- a/tests/xfs/270.out +++ b/tests/xfs/270.out @@ -2,3 +2,5 @@ QA output created by 270 rw mount test ro mount test rw remount test +rw mount test +ro mount test ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features 2023-08-29 23:09 [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features Darrick J. Wong 2023-08-29 23:09 ` [PATCH 1/2] xfs/270: actually test file readability Darrick J. Wong 2023-08-29 23:09 ` [PATCH 2/2] xfs/270: actually test log recovery with unknown rocompat features Darrick J. Wong @ 2023-09-17 16:44 ` Zorro Lang 2023-09-18 15:17 ` Darrick J. Wong 2 siblings, 1 reply; 10+ messages in thread From: Zorro Lang @ 2023-09-17 16:44 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests Hi Darrick, Hi Darrick, Do these two changes cover new xfs bug fix? I hit xfs/270 fails on latest upstream mainline kernel testing, e.g. [1][2]. But the xfs/270 always test passed on my side before. Thanks, Zorro [1] --- /dev/fd/63 2023-09-16 20:33:38.373115157 -0400 +++ xfs/270.out.bad 2023-09-16 20:33:37.959115275 -0400 @@ -1,6 +1,9 @@ QA output created by 270 rw mount test ro mount test +cat: /mnt/xfstests/scratch/testfile: Input/output error rw remount test rw mount test ro mount test +ro mount test failed +(see /var/lib/xfstests/results//xfs/270.full for details) [2] [57829.436755] run fstests xfs/270 at 2023-09-16 20:33:31 [57832.373519] XFS (vda3): Mounting V5 Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb [57832.398699] XFS (vda3): Ending clean mount [57832.551895] XFS (vda3): Unmounting Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb [57832.909487] XFS (vda3): Mounting V5 Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc [57832.934603] XFS (vda3): Ending clean mount [57832.963287] XFS (vda3): User initiated shutdown received. [57832.967959] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. [57832.974742] XFS (vda3): Please unmount the filesystem and rectify the problem(s) [57833.002720] XFS (vda3): Unmounting Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc [57833.376473] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 [57833.403380] XFS (vda3): Ending clean mount [57833.443719] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 [57833.516114] [TTM] Buffer eviction failed [57833.519490] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001) [57833.523184] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO [57833.955746] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. [57833.960383] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. [57833.964353] XFS (vda3): Filesystem can only be safely mounted read only. [57833.967997] XFS (vda3): SB validate failed with error -22. [57834.039637] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. [57834.046289] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 [57834.075451] XFS (vda3): Ending clean mount [57834.082359] XFS (vda3): Corruption detected in superblock read-only compatible features (0x80000000)! [57834.086487] XFS (vda3): Metadata corruption detected at xfs_sb_write_verify+0x11d/0x380 [xfs], xfs_sb block 0x0 [57834.090716] XFS (vda3): Unmount and run xfs_repair [57834.093708] XFS (vda3): First 128 bytes of corrupted metadata buffer: [57834.096953] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 3c 00 00 XFSB.........<.. [57834.100635] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [57834.104330] 00000020: 80 1b 25 ca ac c4 42 32 99 55 dc 34 48 68 04 67 ..%...B2.U.4Hh.g [57834.108273] 00000030: 00 00 00 00 00 20 00 06 00 00 00 00 00 00 00 80 ..... .......... [57834.112176] 00000040: 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82 ................ [57834.116043] 00000050: 00 00 00 01 00 0f 00 00 00 00 00 04 00 00 00 00 ................ [57834.119693] 00000060: 00 00 40 00 b4 a5 02 00 02 00 00 08 00 00 00 00 ..@............. [57834.123459] 00000070: 00 00 00 00 00 00 00 00 0c 09 09 03 14 00 00 19 ................ [57834.127159] XFS (vda3): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x50b/0x5f0 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. [57834.134557] XFS (vda3): Please unmount the filesystem and rectify the problem(s) [57834.253491] XFS (vda3): ro->rw transition prohibited on unknown (0x80000000) ro-compat filesystem [57834.281980] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 [57834.655120] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 [57834.682299] XFS (vda3): Ending clean mount [57834.723067] XFS (vda3): User initiated shutdown received. [57834.730954] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. [57834.738710] XFS (vda3): Please unmount the filesystem and rectify the problem(s) [57834.768607] XFS (vda3): Unmounting Filesystem 29902d21-a146-4095-8747-562850110ff6 [57835.270472] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. [57835.275655] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. [57835.279434] XFS (vda3): Filesystem can only be safely mounted read only. [57835.283124] XFS (vda3): SB validate failed with error -22. [57835.357336] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. [57835.369968] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 [57835.398595] XFS (vda3): Starting recovery (logdev: internal) [57835.410361] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. [57835.414593] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. [57835.418495] XFS (vda3): Filesystem can only be safely mounted read only. [57835.422152] XFS (vda3): metadata I/O error in "xlog_do_recover+0x3e8/0x540 [xfs]" at daddr 0x0 len 1 error 22 [57835.426851] XFS: Assertion failed: 0, file: fs/xfs/xfs_log_recover.c, line: 3321 [57835.430765] ------------[ cut here ]------------ [57835.434014] WARNING: CPU: 2 PID: 3426232 at fs/xfs/xfs_message.c:104 assfail+0x54/0x70 [xfs] [57835.438380] Modules linked in: ext2 overlay dm_zero dm_log_writes dm_thin_pool dm_persistent_data dm_bio_prison sd_mod t10_pi sg dm_snapshot dm_bufio ext4 mbcache jbd2 loop dm_flakey dm_mod tls rfkill intel_rapl_msr sunrpc intel_rapl_common intel_uncore_frequency_common isst_if_common nfit snd_hda_codec_generic ledtrig_audio kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_seq snd_seq_device snd_pcm qxl snd_timer drm_ttm_helper ttm pcspkr virtio_balloon snd drm_kms_helper soundcore i2c_piix4 joydev drm fuse xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix virtio_console virtio_net net_failover failover virtio_blk libata serio_raw [last unloaded: scsi_debug] [57835.467794] CPU: 2 PID: 3426232 Comm: mount Kdump: loaded Tainted: G W 6.6.0-rc1+ #1 [57835.472233] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [57835.475946] RIP: 0010:assfail+0x54/0x70 [xfs] [57835.479708] Code: c0 48 ba 00 00 00 00 00 fc ff df 48 89 c1 83 e0 07 48 c1 e9 03 0f b6 14 11 38 c2 7f 04 84 d2 75 10 80 3d 6e 3f 1c 00 00 75 15 <0f> 0b c3 cc cc cc cc 48 c7 c7 10 47 99 c0 e8 79 fd 34 df eb e2 0f [57835.489526] RSP: 0018:ffffc9000293f918 EFLAGS: 00010246 [57835.493381] RAX: 0000000000000000 RBX: ffff88817b73a000 RCX: 1ffffffff81328e2 [57835.497512] RDX: 0000000000000004 RSI: ffffc9000293f680 RDI: ffffffffc0afd780 [57835.501588] RBP: ffff8881964ea000 R08: 00000000ffffffea R09: fffff52000527ec1 [57835.505701] R10: ffffc9000293f60f R11: fffffffffff6c2b0 R12: 00000000ffffffea [57835.509789] R13: ffff88817b73a0e0 R14: ffff8881b927ba80 R15: On Tue, Aug 29, 2023 at 04:09:48PM -0700, Darrick J. Wong wrote: > Hi all, > > Dave pointed out some failures in xfs/270 when he upgraded Debian > unstable and util-linux started using the new mount apis. Upon further > inquiry I noticed that XFS is quite a hot mess when it encounters a > filesystem with unrecognized rocompat bits set in the superblock. > > Whereas we used to allow readonly mounts under these conditions, a > change to the sb write verifier several years ago resulted in the > filesystem going down immediately because the post-mount log cleaning > writes the superblock, which trips the sb write verifier on the > unrecognized rocompat bit. I made the observation that the ROCOMPAT > features RMAPBT and REFLINK both protect new log intent item types, > which means that we actually cannot support recovering the log if we > don't recognize all the rocompat bits. > > Therefore -- fix inode inactivation to work when we're recovering the > log, disallow recovery when there's unrecognized rocompat bits, and > don't clean the log if doing so would trip the rocompat checks. > > v2: change direction of series to allow log recovery on ro mounts > > If you're going to start using this code, I strongly recommend pulling > from my git trees, which are linked below. > > This has been running on the djcloud for months with no problems. Enjoy! > Comments and questions are, as always, welcome. > > --D > > fstests git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts > --- > tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++--------------- > tests/xfs/270.out | 2 + > 2 files changed, 59 insertions(+), 23 deletions(-) > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features 2023-09-17 16:44 ` [PATCHSET v2 0/2] fstests: fix ro mounting " Zorro Lang @ 2023-09-18 15:17 ` Darrick J. Wong 2023-09-18 16:44 ` Zorro Lang 0 siblings, 1 reply; 10+ messages in thread From: Darrick J. Wong @ 2023-09-18 15:17 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Mon, Sep 18, 2023 at 12:44:40AM +0800, Zorro Lang wrote: > Hi Darrick, > > Hi Darrick, > > Do these two changes cover new xfs bug fix? I hit xfs/270 fails on > latest upstream mainline kernel testing, e.g. [1][2]. But the xfs/270 > always test passed on my side before. Yes, the changes in xfs/270 reflect the corrections to the ro mounting behavior in this fix series: https://lore.kernel.org/linux-xfs/169454023347.3411463.3256623887249978768.stg-ugh@frogsfrogsfrogs/T/#u The ro mount behavior regressed some 7-8 years ago; this test was written to test the broken fix; and we didn't notice any of this until 6.1 added the incore unlinked list. xfs/270 didn't check the usability of the mounted ro filesystem, so nobody noticed that the mount failed when it should have succeeded. (That's the first + line) The second diff here checks what happens when you try to ro mount with a dirty log. --D > Thanks, > Zorro > > [1] > --- /dev/fd/63 2023-09-16 20:33:38.373115157 -0400 > +++ xfs/270.out.bad 2023-09-16 20:33:37.959115275 -0400 > @@ -1,6 +1,9 @@ > QA output created by 270 > rw mount test > ro mount test > +cat: /mnt/xfstests/scratch/testfile: Input/output error > rw remount test > rw mount test > ro mount test > +ro mount test failed > +(see /var/lib/xfstests/results//xfs/270.full for details) > > [2] > [57829.436755] run fstests xfs/270 at 2023-09-16 20:33:31 > [57832.373519] XFS (vda3): Mounting V5 Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > [57832.398699] XFS (vda3): Ending clean mount > [57832.551895] XFS (vda3): Unmounting Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > [57832.909487] XFS (vda3): Mounting V5 Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > [57832.934603] XFS (vda3): Ending clean mount > [57832.963287] XFS (vda3): User initiated shutdown received. > [57832.967959] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > [57832.974742] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > [57833.002720] XFS (vda3): Unmounting Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > [57833.376473] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > [57833.403380] XFS (vda3): Ending clean mount > [57833.443719] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > [57833.516114] [TTM] Buffer eviction failed > [57833.519490] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001) > [57833.523184] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO > [57833.955746] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > [57833.960383] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > [57833.964353] XFS (vda3): Filesystem can only be safely mounted read only. > [57833.967997] XFS (vda3): SB validate failed with error -22. > [57834.039637] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > [57834.046289] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > [57834.075451] XFS (vda3): Ending clean mount > [57834.082359] XFS (vda3): Corruption detected in superblock read-only compatible features (0x80000000)! > [57834.086487] XFS (vda3): Metadata corruption detected at xfs_sb_write_verify+0x11d/0x380 [xfs], xfs_sb block 0x0 > [57834.090716] XFS (vda3): Unmount and run xfs_repair > [57834.093708] XFS (vda3): First 128 bytes of corrupted metadata buffer: > [57834.096953] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 3c 00 00 XFSB.........<.. > [57834.100635] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [57834.104330] 00000020: 80 1b 25 ca ac c4 42 32 99 55 dc 34 48 68 04 67 ..%...B2.U.4Hh.g > [57834.108273] 00000030: 00 00 00 00 00 20 00 06 00 00 00 00 00 00 00 80 ..... .......... > [57834.112176] 00000040: 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82 ................ > [57834.116043] 00000050: 00 00 00 01 00 0f 00 00 00 00 00 04 00 00 00 00 ................ > [57834.119693] 00000060: 00 00 40 00 b4 a5 02 00 02 00 00 08 00 00 00 00 ..@............. > [57834.123459] 00000070: 00 00 00 00 00 00 00 00 0c 09 09 03 14 00 00 19 ................ > [57834.127159] XFS (vda3): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x50b/0x5f0 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. > [57834.134557] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > [57834.253491] XFS (vda3): ro->rw transition prohibited on unknown (0x80000000) ro-compat filesystem > [57834.281980] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > [57834.655120] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > [57834.682299] XFS (vda3): Ending clean mount > [57834.723067] XFS (vda3): User initiated shutdown received. > [57834.730954] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > [57834.738710] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > [57834.768607] XFS (vda3): Unmounting Filesystem 29902d21-a146-4095-8747-562850110ff6 > [57835.270472] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > [57835.275655] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > [57835.279434] XFS (vda3): Filesystem can only be safely mounted read only. > [57835.283124] XFS (vda3): SB validate failed with error -22. > [57835.357336] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > [57835.369968] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > [57835.398595] XFS (vda3): Starting recovery (logdev: internal) > [57835.410361] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > [57835.414593] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > [57835.418495] XFS (vda3): Filesystem can only be safely mounted read only. > [57835.422152] XFS (vda3): metadata I/O error in "xlog_do_recover+0x3e8/0x540 [xfs]" at daddr 0x0 len 1 error 22 > [57835.426851] XFS: Assertion failed: 0, file: fs/xfs/xfs_log_recover.c, line: 3321 > [57835.430765] ------------[ cut here ]------------ > [57835.434014] WARNING: CPU: 2 PID: 3426232 at fs/xfs/xfs_message.c:104 assfail+0x54/0x70 [xfs] > [57835.438380] Modules linked in: ext2 overlay dm_zero dm_log_writes dm_thin_pool dm_persistent_data dm_bio_prison sd_mod t10_pi sg dm_snapshot dm_bufio ext4 mbcache jbd2 loop dm_flakey dm_mod tls rfkill intel_rapl_msr sunrpc intel_rapl_common intel_uncore_frequency_common isst_if_common nfit snd_hda_codec_generic ledtrig_audio kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_seq snd_seq_device snd_pcm qxl snd_timer drm_ttm_helper ttm pcspkr virtio_balloon snd drm_kms_helper soundcore i2c_piix4 joydev drm fuse xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix virtio_console virtio_net net_failover failover virtio_blk libata serio_raw [last unloaded: scsi_debug] > [57835.467794] CPU: 2 PID: 3426232 Comm: mount Kdump: loaded Tainted: G W 6.6.0-rc1+ #1 > [57835.472233] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > [57835.475946] RIP: 0010:assfail+0x54/0x70 [xfs] > [57835.479708] Code: c0 48 ba 00 00 00 00 00 fc ff df 48 89 c1 83 e0 07 48 c1 e9 03 0f b6 14 11 38 c2 7f 04 84 d2 75 10 80 3d 6e 3f 1c 00 00 75 15 <0f> 0b c3 cc cc cc cc 48 c7 c7 10 47 99 c0 e8 79 fd 34 df eb e2 0f > [57835.489526] RSP: 0018:ffffc9000293f918 EFLAGS: 00010246 > [57835.493381] RAX: 0000000000000000 RBX: ffff88817b73a000 RCX: 1ffffffff81328e2 > [57835.497512] RDX: 0000000000000004 RSI: ffffc9000293f680 RDI: ffffffffc0afd780 > [57835.501588] RBP: ffff8881964ea000 R08: 00000000ffffffea R09: fffff52000527ec1 > [57835.505701] R10: ffffc9000293f60f R11: fffffffffff6c2b0 R12: 00000000ffffffea > [57835.509789] R13: ffff88817b73a0e0 R14: ffff8881b927ba80 R15: > > > On Tue, Aug 29, 2023 at 04:09:48PM -0700, Darrick J. Wong wrote: > > Hi all, > > > > Dave pointed out some failures in xfs/270 when he upgraded Debian > > unstable and util-linux started using the new mount apis. Upon further > > inquiry I noticed that XFS is quite a hot mess when it encounters a > > filesystem with unrecognized rocompat bits set in the superblock. > > > > Whereas we used to allow readonly mounts under these conditions, a > > change to the sb write verifier several years ago resulted in the > > filesystem going down immediately because the post-mount log cleaning > > writes the superblock, which trips the sb write verifier on the > > unrecognized rocompat bit. I made the observation that the ROCOMPAT > > features RMAPBT and REFLINK both protect new log intent item types, > > which means that we actually cannot support recovering the log if we > > don't recognize all the rocompat bits. > > > > Therefore -- fix inode inactivation to work when we're recovering the > > log, disallow recovery when there's unrecognized rocompat bits, and > > don't clean the log if doing so would trip the rocompat checks. > > > > v2: change direction of series to allow log recovery on ro mounts > > > > If you're going to start using this code, I strongly recommend pulling > > from my git trees, which are linked below. > > > > This has been running on the djcloud for months with no problems. Enjoy! > > Comments and questions are, as always, welcome. > > > > --D > > > > fstests git tree: > > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts > > --- > > tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++--------------- > > tests/xfs/270.out | 2 + > > 2 files changed, 59 insertions(+), 23 deletions(-) > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features 2023-09-18 15:17 ` Darrick J. Wong @ 2023-09-18 16:44 ` Zorro Lang 2023-09-18 18:35 ` Darrick J. Wong 0 siblings, 1 reply; 10+ messages in thread From: Zorro Lang @ 2023-09-18 16:44 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Mon, Sep 18, 2023 at 08:17:56AM -0700, Darrick J. Wong wrote: > On Mon, Sep 18, 2023 at 12:44:40AM +0800, Zorro Lang wrote: > > Hi Darrick, > > > > Hi Darrick, > > > > Do these two changes cover new xfs bug fix? I hit xfs/270 fails on > > latest upstream mainline kernel testing, e.g. [1][2]. But the xfs/270 > > always test passed on my side before. > > Yes, the changes in xfs/270 reflect the corrections to the ro mounting > behavior in this fix series: > https://lore.kernel.org/linux-xfs/169454023347.3411463.3256623887249978768.stg-ugh@frogsfrogsfrogs/T/#u > > The ro mount behavior regressed some 7-8 years ago; this test was > written to test the broken fix; and we didn't notice any of this until > 6.1 added the incore unlinked list. xfs/270 didn't check the usability > of the mounted ro filesystem, so nobody noticed that the mount failed > when it should have succeeded. > > (That's the first + line) > > The second diff here checks what happens when you try to ro mount with a > dirty log. Thanks Darrick, With the "[PATCH 1/2] xfs/270: actually test file readability", the x/270 only fails as [1], but with the "[PATCH 2/2] xfs/270: actually test log recovery with unknown rocompat feature", I hit one more kernel assertion failure as [2]. Are these two kind of failures all as known, and will be fixed by : [GIT PULL 3/8] xfs: fix ro mounting with unknown rocompat features ? Thanks, Zorro > > --D > > > Thanks, > > Zorro > > > > [1] > > --- /dev/fd/63 2023-09-16 20:33:38.373115157 -0400 > > +++ xfs/270.out.bad 2023-09-16 20:33:37.959115275 -0400 > > @@ -1,6 +1,9 @@ > > QA output created by 270 > > rw mount test > > ro mount test > > +cat: /mnt/xfstests/scratch/testfile: Input/output error > > rw remount test > > rw mount test > > ro mount test > > +ro mount test failed > > +(see /var/lib/xfstests/results//xfs/270.full for details) > > > > [2] > > [57829.436755] run fstests xfs/270 at 2023-09-16 20:33:31 > > [57832.373519] XFS (vda3): Mounting V5 Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > [57832.398699] XFS (vda3): Ending clean mount > > [57832.551895] XFS (vda3): Unmounting Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > [57832.909487] XFS (vda3): Mounting V5 Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > [57832.934603] XFS (vda3): Ending clean mount > > [57832.963287] XFS (vda3): User initiated shutdown received. > > [57832.967959] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > [57832.974742] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > [57833.002720] XFS (vda3): Unmounting Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > [57833.376473] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > [57833.403380] XFS (vda3): Ending clean mount > > [57833.443719] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > [57833.516114] [TTM] Buffer eviction failed > > [57833.519490] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001) > > [57833.523184] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO > > [57833.955746] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > [57833.960383] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > [57833.964353] XFS (vda3): Filesystem can only be safely mounted read only. > > [57833.967997] XFS (vda3): SB validate failed with error -22. > > [57834.039637] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > [57834.046289] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > [57834.075451] XFS (vda3): Ending clean mount > > [57834.082359] XFS (vda3): Corruption detected in superblock read-only compatible features (0x80000000)! > > [57834.086487] XFS (vda3): Metadata corruption detected at xfs_sb_write_verify+0x11d/0x380 [xfs], xfs_sb block 0x0 > > [57834.090716] XFS (vda3): Unmount and run xfs_repair > > [57834.093708] XFS (vda3): First 128 bytes of corrupted metadata buffer: > > [57834.096953] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 3c 00 00 XFSB.........<.. > > [57834.100635] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > [57834.104330] 00000020: 80 1b 25 ca ac c4 42 32 99 55 dc 34 48 68 04 67 ..%...B2.U.4Hh.g > > [57834.108273] 00000030: 00 00 00 00 00 20 00 06 00 00 00 00 00 00 00 80 ..... .......... > > [57834.112176] 00000040: 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82 ................ > > [57834.116043] 00000050: 00 00 00 01 00 0f 00 00 00 00 00 04 00 00 00 00 ................ > > [57834.119693] 00000060: 00 00 40 00 b4 a5 02 00 02 00 00 08 00 00 00 00 ..@............. > > [57834.123459] 00000070: 00 00 00 00 00 00 00 00 0c 09 09 03 14 00 00 19 ................ > > [57834.127159] XFS (vda3): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x50b/0x5f0 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. > > [57834.134557] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > [57834.253491] XFS (vda3): ro->rw transition prohibited on unknown (0x80000000) ro-compat filesystem > > [57834.281980] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > [57834.655120] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > [57834.682299] XFS (vda3): Ending clean mount > > [57834.723067] XFS (vda3): User initiated shutdown received. > > [57834.730954] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > [57834.738710] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > [57834.768607] XFS (vda3): Unmounting Filesystem 29902d21-a146-4095-8747-562850110ff6 > > [57835.270472] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > [57835.275655] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > [57835.279434] XFS (vda3): Filesystem can only be safely mounted read only. > > [57835.283124] XFS (vda3): SB validate failed with error -22. > > [57835.357336] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > [57835.369968] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > [57835.398595] XFS (vda3): Starting recovery (logdev: internal) > > [57835.410361] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > [57835.414593] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > [57835.418495] XFS (vda3): Filesystem can only be safely mounted read only. > > [57835.422152] XFS (vda3): metadata I/O error in "xlog_do_recover+0x3e8/0x540 [xfs]" at daddr 0x0 len 1 error 22 > > [57835.426851] XFS: Assertion failed: 0, file: fs/xfs/xfs_log_recover.c, line: 3321 > > [57835.430765] ------------[ cut here ]------------ > > [57835.434014] WARNING: CPU: 2 PID: 3426232 at fs/xfs/xfs_message.c:104 assfail+0x54/0x70 [xfs] > > [57835.438380] Modules linked in: ext2 overlay dm_zero dm_log_writes dm_thin_pool dm_persistent_data dm_bio_prison sd_mod t10_pi sg dm_snapshot dm_bufio ext4 mbcache jbd2 loop dm_flakey dm_mod tls rfkill intel_rapl_msr sunrpc intel_rapl_common intel_uncore_frequency_common isst_if_common nfit snd_hda_codec_generic ledtrig_audio kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_seq snd_seq_device snd_pcm qxl snd_timer drm_ttm_helper ttm pcspkr virtio_balloon snd drm_kms_helper soundcore i2c_piix4 joydev drm fuse xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix virtio_console virtio_net net_failover failover virtio_blk libata serio_raw [last unloaded: scsi_debug] > > [57835.467794] CPU: 2 PID: 3426232 Comm: mount Kdump: loaded Tainted: G W 6.6.0-rc1+ #1 > > [57835.472233] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > > [57835.475946] RIP: 0010:assfail+0x54/0x70 [xfs] > > [57835.479708] Code: c0 48 ba 00 00 00 00 00 fc ff df 48 89 c1 83 e0 07 48 c1 e9 03 0f b6 14 11 38 c2 7f 04 84 d2 75 10 80 3d 6e 3f 1c 00 00 75 15 <0f> 0b c3 cc cc cc cc 48 c7 c7 10 47 99 c0 e8 79 fd 34 df eb e2 0f > > [57835.489526] RSP: 0018:ffffc9000293f918 EFLAGS: 00010246 > > [57835.493381] RAX: 0000000000000000 RBX: ffff88817b73a000 RCX: 1ffffffff81328e2 > > [57835.497512] RDX: 0000000000000004 RSI: ffffc9000293f680 RDI: ffffffffc0afd780 > > [57835.501588] RBP: ffff8881964ea000 R08: 00000000ffffffea R09: fffff52000527ec1 > > [57835.505701] R10: ffffc9000293f60f R11: fffffffffff6c2b0 R12: 00000000ffffffea > > [57835.509789] R13: ffff88817b73a0e0 R14: ffff8881b927ba80 R15: > > > > > > On Tue, Aug 29, 2023 at 04:09:48PM -0700, Darrick J. Wong wrote: > > > Hi all, > > > > > > Dave pointed out some failures in xfs/270 when he upgraded Debian > > > unstable and util-linux started using the new mount apis. Upon further > > > inquiry I noticed that XFS is quite a hot mess when it encounters a > > > filesystem with unrecognized rocompat bits set in the superblock. > > > > > > Whereas we used to allow readonly mounts under these conditions, a > > > change to the sb write verifier several years ago resulted in the > > > filesystem going down immediately because the post-mount log cleaning > > > writes the superblock, which trips the sb write verifier on the > > > unrecognized rocompat bit. I made the observation that the ROCOMPAT > > > features RMAPBT and REFLINK both protect new log intent item types, > > > which means that we actually cannot support recovering the log if we > > > don't recognize all the rocompat bits. > > > > > > Therefore -- fix inode inactivation to work when we're recovering the > > > log, disallow recovery when there's unrecognized rocompat bits, and > > > don't clean the log if doing so would trip the rocompat checks. > > > > > > v2: change direction of series to allow log recovery on ro mounts > > > > > > If you're going to start using this code, I strongly recommend pulling > > > from my git trees, which are linked below. > > > > > > This has been running on the djcloud for months with no problems. Enjoy! > > > Comments and questions are, as always, welcome. > > > > > > --D > > > > > > fstests git tree: > > > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts > > > --- > > > tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++--------------- > > > tests/xfs/270.out | 2 + > > > 2 files changed, 59 insertions(+), 23 deletions(-) > > > > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features 2023-09-18 16:44 ` Zorro Lang @ 2023-09-18 18:35 ` Darrick J. Wong 2023-09-19 2:57 ` Zorro Lang 0 siblings, 1 reply; 10+ messages in thread From: Darrick J. Wong @ 2023-09-18 18:35 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Tue, Sep 19, 2023 at 12:44:49AM +0800, Zorro Lang wrote: > On Mon, Sep 18, 2023 at 08:17:56AM -0700, Darrick J. Wong wrote: > > On Mon, Sep 18, 2023 at 12:44:40AM +0800, Zorro Lang wrote: > > > Hi Darrick, > > > > > > Hi Darrick, > > > > > > Do these two changes cover new xfs bug fix? I hit xfs/270 fails on > > > latest upstream mainline kernel testing, e.g. [1][2]. But the xfs/270 > > > always test passed on my side before. > > > > Yes, the changes in xfs/270 reflect the corrections to the ro mounting > > behavior in this fix series: > > https://lore.kernel.org/linux-xfs/169454023347.3411463.3256623887249978768.stg-ugh@frogsfrogsfrogs/T/#u > > > > The ro mount behavior regressed some 7-8 years ago; this test was > > written to test the broken fix; and we didn't notice any of this until > > 6.1 added the incore unlinked list. xfs/270 didn't check the usability > > of the mounted ro filesystem, so nobody noticed that the mount failed > > when it should have succeeded. > > > > (That's the first + line) > > > > The second diff here checks what happens when you try to ro mount with a > > dirty log. > > Thanks Darrick, > > With the "[PATCH 1/2] xfs/270: actually test file readability", the x/270 only > fails as [1], but with the "[PATCH 2/2] xfs/270: actually test log recovery with > unknown rocompat feature", I hit one more kernel assertion failure as [2]. > > Are these two kind of failures all as known, and will be fixed by : > [GIT PULL 3/8] xfs: fix ro mounting with unknown rocompat features > ? Yes. That second assertion is log recovery complaining that the fs shut down during recovery; and the reason the log shut down during recovery is that someone tried to write a superblock with unknown rocompat bits and triggered the verifier. The verifier failure is addressed by "xfs: fix log recovery when unknown rocompat bits are set". --D > Thanks, > Zorro > > > > > > --D > > > > > Thanks, > > > Zorro > > > > > > [1] > > > --- /dev/fd/63 2023-09-16 20:33:38.373115157 -0400 > > > +++ xfs/270.out.bad 2023-09-16 20:33:37.959115275 -0400 > > > @@ -1,6 +1,9 @@ > > > QA output created by 270 > > > rw mount test > > > ro mount test > > > +cat: /mnt/xfstests/scratch/testfile: Input/output error > > > rw remount test > > > rw mount test > > > ro mount test > > > +ro mount test failed > > > +(see /var/lib/xfstests/results//xfs/270.full for details) > > > > > > [2] > > > [57829.436755] run fstests xfs/270 at 2023-09-16 20:33:31 > > > [57832.373519] XFS (vda3): Mounting V5 Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > > [57832.398699] XFS (vda3): Ending clean mount > > > [57832.551895] XFS (vda3): Unmounting Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > > [57832.909487] XFS (vda3): Mounting V5 Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > > [57832.934603] XFS (vda3): Ending clean mount > > > [57832.963287] XFS (vda3): User initiated shutdown received. > > > [57832.967959] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > > [57832.974742] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > [57833.002720] XFS (vda3): Unmounting Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > > [57833.376473] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > [57833.403380] XFS (vda3): Ending clean mount > > > [57833.443719] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > [57833.516114] [TTM] Buffer eviction failed > > > [57833.519490] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001) > > > [57833.523184] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO > > > [57833.955746] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > [57833.960383] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > [57833.964353] XFS (vda3): Filesystem can only be safely mounted read only. > > > [57833.967997] XFS (vda3): SB validate failed with error -22. > > > [57834.039637] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > [57834.046289] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > [57834.075451] XFS (vda3): Ending clean mount > > > [57834.082359] XFS (vda3): Corruption detected in superblock read-only compatible features (0x80000000)! > > > [57834.086487] XFS (vda3): Metadata corruption detected at xfs_sb_write_verify+0x11d/0x380 [xfs], xfs_sb block 0x0 > > > [57834.090716] XFS (vda3): Unmount and run xfs_repair > > > [57834.093708] XFS (vda3): First 128 bytes of corrupted metadata buffer: > > > [57834.096953] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 3c 00 00 XFSB.........<.. > > > [57834.100635] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > > [57834.104330] 00000020: 80 1b 25 ca ac c4 42 32 99 55 dc 34 48 68 04 67 ..%...B2.U.4Hh.g > > > [57834.108273] 00000030: 00 00 00 00 00 20 00 06 00 00 00 00 00 00 00 80 ..... .......... > > > [57834.112176] 00000040: 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82 ................ > > > [57834.116043] 00000050: 00 00 00 01 00 0f 00 00 00 00 00 04 00 00 00 00 ................ > > > [57834.119693] 00000060: 00 00 40 00 b4 a5 02 00 02 00 00 08 00 00 00 00 ..@............. > > > [57834.123459] 00000070: 00 00 00 00 00 00 00 00 0c 09 09 03 14 00 00 19 ................ > > > [57834.127159] XFS (vda3): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x50b/0x5f0 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. > > > [57834.134557] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > [57834.253491] XFS (vda3): ro->rw transition prohibited on unknown (0x80000000) ro-compat filesystem > > > [57834.281980] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > [57834.655120] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > [57834.682299] XFS (vda3): Ending clean mount > > > [57834.723067] XFS (vda3): User initiated shutdown received. > > > [57834.730954] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > > [57834.738710] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > [57834.768607] XFS (vda3): Unmounting Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > [57835.270472] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > [57835.275655] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > [57835.279434] XFS (vda3): Filesystem can only be safely mounted read only. > > > [57835.283124] XFS (vda3): SB validate failed with error -22. > > > [57835.357336] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > [57835.369968] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > [57835.398595] XFS (vda3): Starting recovery (logdev: internal) > > > [57835.410361] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > [57835.414593] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > [57835.418495] XFS (vda3): Filesystem can only be safely mounted read only. > > > [57835.422152] XFS (vda3): metadata I/O error in "xlog_do_recover+0x3e8/0x540 [xfs]" at daddr 0x0 len 1 error 22 > > > [57835.426851] XFS: Assertion failed: 0, file: fs/xfs/xfs_log_recover.c, line: 3321 > > > [57835.430765] ------------[ cut here ]------------ > > > [57835.434014] WARNING: CPU: 2 PID: 3426232 at fs/xfs/xfs_message.c:104 assfail+0x54/0x70 [xfs] > > > [57835.438380] Modules linked in: ext2 overlay dm_zero dm_log_writes dm_thin_pool dm_persistent_data dm_bio_prison sd_mod t10_pi sg dm_snapshot dm_bufio ext4 mbcache jbd2 loop dm_flakey dm_mod tls rfkill intel_rapl_msr sunrpc intel_rapl_common intel_uncore_frequency_common isst_if_common nfit snd_hda_codec_generic ledtrig_audio kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_seq snd_seq_device snd_pcm qxl snd_timer drm_ttm_helper ttm pcspkr virtio_balloon snd drm_kms_helper soundcore i2c_piix4 joydev drm fuse xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix virtio_console virtio_net net_failover failover virtio_blk libata serio_raw [last unloaded: scsi_debug] > > > [57835.467794] CPU: 2 PID: 3426232 Comm: mount Kdump: loaded Tainted: G W 6.6.0-rc1+ #1 > > > [57835.472233] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > > > [57835.475946] RIP: 0010:assfail+0x54/0x70 [xfs] > > > [57835.479708] Code: c0 48 ba 00 00 00 00 00 fc ff df 48 89 c1 83 e0 07 48 c1 e9 03 0f b6 14 11 38 c2 7f 04 84 d2 75 10 80 3d 6e 3f 1c 00 00 75 15 <0f> 0b c3 cc cc cc cc 48 c7 c7 10 47 99 c0 e8 79 fd 34 df eb e2 0f > > > [57835.489526] RSP: 0018:ffffc9000293f918 EFLAGS: 00010246 > > > [57835.493381] RAX: 0000000000000000 RBX: ffff88817b73a000 RCX: 1ffffffff81328e2 > > > [57835.497512] RDX: 0000000000000004 RSI: ffffc9000293f680 RDI: ffffffffc0afd780 > > > [57835.501588] RBP: ffff8881964ea000 R08: 00000000ffffffea R09: fffff52000527ec1 > > > [57835.505701] R10: ffffc9000293f60f R11: fffffffffff6c2b0 R12: 00000000ffffffea > > > [57835.509789] R13: ffff88817b73a0e0 R14: ffff8881b927ba80 R15: > > > > > > > > > On Tue, Aug 29, 2023 at 04:09:48PM -0700, Darrick J. Wong wrote: > > > > Hi all, > > > > > > > > Dave pointed out some failures in xfs/270 when he upgraded Debian > > > > unstable and util-linux started using the new mount apis. Upon further > > > > inquiry I noticed that XFS is quite a hot mess when it encounters a > > > > filesystem with unrecognized rocompat bits set in the superblock. > > > > > > > > Whereas we used to allow readonly mounts under these conditions, a > > > > change to the sb write verifier several years ago resulted in the > > > > filesystem going down immediately because the post-mount log cleaning > > > > writes the superblock, which trips the sb write verifier on the > > > > unrecognized rocompat bit. I made the observation that the ROCOMPAT > > > > features RMAPBT and REFLINK both protect new log intent item types, > > > > which means that we actually cannot support recovering the log if we > > > > don't recognize all the rocompat bits. > > > > > > > > Therefore -- fix inode inactivation to work when we're recovering the > > > > log, disallow recovery when there's unrecognized rocompat bits, and > > > > don't clean the log if doing so would trip the rocompat checks. > > > > > > > > v2: change direction of series to allow log recovery on ro mounts > > > > > > > > If you're going to start using this code, I strongly recommend pulling > > > > from my git trees, which are linked below. > > > > > > > > This has been running on the djcloud for months with no problems. Enjoy! > > > > Comments and questions are, as always, welcome. > > > > > > > > --D > > > > > > > > fstests git tree: > > > > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts > > > > --- > > > > tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++--------------- > > > > tests/xfs/270.out | 2 + > > > > 2 files changed, 59 insertions(+), 23 deletions(-) > > > > > > > > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features 2023-09-18 18:35 ` Darrick J. Wong @ 2023-09-19 2:57 ` Zorro Lang 2023-09-19 3:06 ` Darrick J. Wong 0 siblings, 1 reply; 10+ messages in thread From: Zorro Lang @ 2023-09-19 2:57 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Mon, Sep 18, 2023 at 11:35:06AM -0700, Darrick J. Wong wrote: > On Tue, Sep 19, 2023 at 12:44:49AM +0800, Zorro Lang wrote: > > On Mon, Sep 18, 2023 at 08:17:56AM -0700, Darrick J. Wong wrote: > > > On Mon, Sep 18, 2023 at 12:44:40AM +0800, Zorro Lang wrote: > > > > Hi Darrick, > > > > > > > > Hi Darrick, > > > > > > > > Do these two changes cover new xfs bug fix? I hit xfs/270 fails on > > > > latest upstream mainline kernel testing, e.g. [1][2]. But the xfs/270 > > > > always test passed on my side before. > > > > > > Yes, the changes in xfs/270 reflect the corrections to the ro mounting > > > behavior in this fix series: > > > https://lore.kernel.org/linux-xfs/169454023347.3411463.3256623887249978768.stg-ugh@frogsfrogsfrogs/T/#u > > > > > > The ro mount behavior regressed some 7-8 years ago; this test was > > > written to test the broken fix; and we didn't notice any of this until > > > 6.1 added the incore unlinked list. xfs/270 didn't check the usability > > > of the mounted ro filesystem, so nobody noticed that the mount failed > > > when it should have succeeded. > > > > > > (That's the first + line) > > > > > > The second diff here checks what happens when you try to ro mount with a > > > dirty log. > > > > Thanks Darrick, > > > > With the "[PATCH 1/2] xfs/270: actually test file readability", the x/270 only > > fails as [1], but with the "[PATCH 2/2] xfs/270: actually test log recovery with > > unknown rocompat feature", I hit one more kernel assertion failure as [2]. > > > > Are these two kind of failures all as known, and will be fixed by : > > [GIT PULL 3/8] xfs: fix ro mounting with unknown rocompat features > > ? > > Yes. That second assertion is log recovery complaining that the fs shut > down during recovery; and the reason the log shut down during recovery > is that someone tried to write a superblock with unknown rocompat bits > and triggered the verifier. The verifier failure is addressed by "xfs: > fix log recovery when unknown rocompat bits are set". Thanks, if you don't mind, I'll add _fixed_by_kernel_commit xxxxxxxxxxxx \ "xfs: fix log recovery when unknown rocompat bits are set" in the 1st patch, if the two patches point to a same one fix. Is that good to you? Thanks, Zorro > > --D > > > Thanks, > > Zorro > > > > > > > > > > --D > > > > > > > Thanks, > > > > Zorro > > > > > > > > [1] > > > > --- /dev/fd/63 2023-09-16 20:33:38.373115157 -0400 > > > > +++ xfs/270.out.bad 2023-09-16 20:33:37.959115275 -0400 > > > > @@ -1,6 +1,9 @@ > > > > QA output created by 270 > > > > rw mount test > > > > ro mount test > > > > +cat: /mnt/xfstests/scratch/testfile: Input/output error > > > > rw remount test > > > > rw mount test > > > > ro mount test > > > > +ro mount test failed > > > > +(see /var/lib/xfstests/results//xfs/270.full for details) > > > > > > > > [2] > > > > [57829.436755] run fstests xfs/270 at 2023-09-16 20:33:31 > > > > [57832.373519] XFS (vda3): Mounting V5 Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > > > [57832.398699] XFS (vda3): Ending clean mount > > > > [57832.551895] XFS (vda3): Unmounting Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > > > [57832.909487] XFS (vda3): Mounting V5 Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > > > [57832.934603] XFS (vda3): Ending clean mount > > > > [57832.963287] XFS (vda3): User initiated shutdown received. > > > > [57832.967959] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > > > [57832.974742] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > > [57833.002720] XFS (vda3): Unmounting Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > > > [57833.376473] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > [57833.403380] XFS (vda3): Ending clean mount > > > > [57833.443719] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > [57833.516114] [TTM] Buffer eviction failed > > > > [57833.519490] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001) > > > > [57833.523184] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO > > > > [57833.955746] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > [57833.960383] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > > [57833.964353] XFS (vda3): Filesystem can only be safely mounted read only. > > > > [57833.967997] XFS (vda3): SB validate failed with error -22. > > > > [57834.039637] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > [57834.046289] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > [57834.075451] XFS (vda3): Ending clean mount > > > > [57834.082359] XFS (vda3): Corruption detected in superblock read-only compatible features (0x80000000)! > > > > [57834.086487] XFS (vda3): Metadata corruption detected at xfs_sb_write_verify+0x11d/0x380 [xfs], xfs_sb block 0x0 > > > > [57834.090716] XFS (vda3): Unmount and run xfs_repair > > > > [57834.093708] XFS (vda3): First 128 bytes of corrupted metadata buffer: > > > > [57834.096953] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 3c 00 00 XFSB.........<.. > > > > [57834.100635] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > > > [57834.104330] 00000020: 80 1b 25 ca ac c4 42 32 99 55 dc 34 48 68 04 67 ..%...B2.U.4Hh.g > > > > [57834.108273] 00000030: 00 00 00 00 00 20 00 06 00 00 00 00 00 00 00 80 ..... .......... > > > > [57834.112176] 00000040: 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82 ................ > > > > [57834.116043] 00000050: 00 00 00 01 00 0f 00 00 00 00 00 04 00 00 00 00 ................ > > > > [57834.119693] 00000060: 00 00 40 00 b4 a5 02 00 02 00 00 08 00 00 00 00 ..@............. > > > > [57834.123459] 00000070: 00 00 00 00 00 00 00 00 0c 09 09 03 14 00 00 19 ................ > > > > [57834.127159] XFS (vda3): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x50b/0x5f0 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. > > > > [57834.134557] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > > [57834.253491] XFS (vda3): ro->rw transition prohibited on unknown (0x80000000) ro-compat filesystem > > > > [57834.281980] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > [57834.655120] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > > [57834.682299] XFS (vda3): Ending clean mount > > > > [57834.723067] XFS (vda3): User initiated shutdown received. > > > > [57834.730954] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > > > [57834.738710] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > > [57834.768607] XFS (vda3): Unmounting Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > > [57835.270472] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > [57835.275655] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > > [57835.279434] XFS (vda3): Filesystem can only be safely mounted read only. > > > > [57835.283124] XFS (vda3): SB validate failed with error -22. > > > > [57835.357336] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > [57835.369968] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > > [57835.398595] XFS (vda3): Starting recovery (logdev: internal) > > > > [57835.410361] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > [57835.414593] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > > [57835.418495] XFS (vda3): Filesystem can only be safely mounted read only. > > > > [57835.422152] XFS (vda3): metadata I/O error in "xlog_do_recover+0x3e8/0x540 [xfs]" at daddr 0x0 len 1 error 22 > > > > [57835.426851] XFS: Assertion failed: 0, file: fs/xfs/xfs_log_recover.c, line: 3321 > > > > [57835.430765] ------------[ cut here ]------------ > > > > [57835.434014] WARNING: CPU: 2 PID: 3426232 at fs/xfs/xfs_message.c:104 assfail+0x54/0x70 [xfs] > > > > [57835.438380] Modules linked in: ext2 overlay dm_zero dm_log_writes dm_thin_pool dm_persistent_data dm_bio_prison sd_mod t10_pi sg dm_snapshot dm_bufio ext4 mbcache jbd2 loop dm_flakey dm_mod tls rfkill intel_rapl_msr sunrpc intel_rapl_common intel_uncore_frequency_common isst_if_common nfit snd_hda_codec_generic ledtrig_audio kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_seq snd_seq_device snd_pcm qxl snd_timer drm_ttm_helper ttm pcspkr virtio_balloon snd drm_kms_helper soundcore i2c_piix4 joydev drm fuse xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix virtio_console virtio_net net_failover failover virtio_blk libata serio_raw [last unloaded: scsi_debug] > > > > [57835.467794] CPU: 2 PID: 3426232 Comm: mount Kdump: loaded Tainted: G W 6.6.0-rc1+ #1 > > > > [57835.472233] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > > > > [57835.475946] RIP: 0010:assfail+0x54/0x70 [xfs] > > > > [57835.479708] Code: c0 48 ba 00 00 00 00 00 fc ff df 48 89 c1 83 e0 07 48 c1 e9 03 0f b6 14 11 38 c2 7f 04 84 d2 75 10 80 3d 6e 3f 1c 00 00 75 15 <0f> 0b c3 cc cc cc cc 48 c7 c7 10 47 99 c0 e8 79 fd 34 df eb e2 0f > > > > [57835.489526] RSP: 0018:ffffc9000293f918 EFLAGS: 00010246 > > > > [57835.493381] RAX: 0000000000000000 RBX: ffff88817b73a000 RCX: 1ffffffff81328e2 > > > > [57835.497512] RDX: 0000000000000004 RSI: ffffc9000293f680 RDI: ffffffffc0afd780 > > > > [57835.501588] RBP: ffff8881964ea000 R08: 00000000ffffffea R09: fffff52000527ec1 > > > > [57835.505701] R10: ffffc9000293f60f R11: fffffffffff6c2b0 R12: 00000000ffffffea > > > > [57835.509789] R13: ffff88817b73a0e0 R14: ffff8881b927ba80 R15: > > > > > > > > > > > > On Tue, Aug 29, 2023 at 04:09:48PM -0700, Darrick J. Wong wrote: > > > > > Hi all, > > > > > > > > > > Dave pointed out some failures in xfs/270 when he upgraded Debian > > > > > unstable and util-linux started using the new mount apis. Upon further > > > > > inquiry I noticed that XFS is quite a hot mess when it encounters a > > > > > filesystem with unrecognized rocompat bits set in the superblock. > > > > > > > > > > Whereas we used to allow readonly mounts under these conditions, a > > > > > change to the sb write verifier several years ago resulted in the > > > > > filesystem going down immediately because the post-mount log cleaning > > > > > writes the superblock, which trips the sb write verifier on the > > > > > unrecognized rocompat bit. I made the observation that the ROCOMPAT > > > > > features RMAPBT and REFLINK both protect new log intent item types, > > > > > which means that we actually cannot support recovering the log if we > > > > > don't recognize all the rocompat bits. > > > > > > > > > > Therefore -- fix inode inactivation to work when we're recovering the > > > > > log, disallow recovery when there's unrecognized rocompat bits, and > > > > > don't clean the log if doing so would trip the rocompat checks. > > > > > > > > > > v2: change direction of series to allow log recovery on ro mounts > > > > > > > > > > If you're going to start using this code, I strongly recommend pulling > > > > > from my git trees, which are linked below. > > > > > > > > > > This has been running on the djcloud for months with no problems. Enjoy! > > > > > Comments and questions are, as always, welcome. > > > > > > > > > > --D > > > > > > > > > > fstests git tree: > > > > > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts > > > > > --- > > > > > tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++--------------- > > > > > tests/xfs/270.out | 2 + > > > > > 2 files changed, 59 insertions(+), 23 deletions(-) > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features 2023-09-19 2:57 ` Zorro Lang @ 2023-09-19 3:06 ` Darrick J. Wong 0 siblings, 0 replies; 10+ messages in thread From: Darrick J. Wong @ 2023-09-19 3:06 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Tue, Sep 19, 2023 at 10:57:49AM +0800, Zorro Lang wrote: > On Mon, Sep 18, 2023 at 11:35:06AM -0700, Darrick J. Wong wrote: > > On Tue, Sep 19, 2023 at 12:44:49AM +0800, Zorro Lang wrote: > > > On Mon, Sep 18, 2023 at 08:17:56AM -0700, Darrick J. Wong wrote: > > > > On Mon, Sep 18, 2023 at 12:44:40AM +0800, Zorro Lang wrote: > > > > > Hi Darrick, > > > > > > > > > > Hi Darrick, > > > > > > > > > > Do these two changes cover new xfs bug fix? I hit xfs/270 fails on > > > > > latest upstream mainline kernel testing, e.g. [1][2]. But the xfs/270 > > > > > always test passed on my side before. > > > > > > > > Yes, the changes in xfs/270 reflect the corrections to the ro mounting > > > > behavior in this fix series: > > > > https://lore.kernel.org/linux-xfs/169454023347.3411463.3256623887249978768.stg-ugh@frogsfrogsfrogs/T/#u > > > > > > > > The ro mount behavior regressed some 7-8 years ago; this test was > > > > written to test the broken fix; and we didn't notice any of this until > > > > 6.1 added the incore unlinked list. xfs/270 didn't check the usability > > > > of the mounted ro filesystem, so nobody noticed that the mount failed > > > > when it should have succeeded. > > > > > > > > (That's the first + line) > > > > > > > > The second diff here checks what happens when you try to ro mount with a > > > > dirty log. > > > > > > Thanks Darrick, > > > > > > With the "[PATCH 1/2] xfs/270: actually test file readability", the x/270 only > > > fails as [1], but with the "[PATCH 2/2] xfs/270: actually test log recovery with > > > unknown rocompat feature", I hit one more kernel assertion failure as [2]. > > > > > > Are these two kind of failures all as known, and will be fixed by : > > > [GIT PULL 3/8] xfs: fix ro mounting with unknown rocompat features > > > ? > > > > Yes. That second assertion is log recovery complaining that the fs shut > > down during recovery; and the reason the log shut down during recovery > > is that someone tried to write a superblock with unknown rocompat bits > > and triggered the verifier. The verifier failure is addressed by "xfs: > > fix log recovery when unknown rocompat bits are set". > > Thanks, if you don't mind, I'll add > > _fixed_by_kernel_commit xxxxxxxxxxxx \ > "xfs: fix log recovery when unknown rocompat bits are set" > > in the 1st patch, if the two patches point to a same one fix. Is that > good to you? Yes, thank you! --D > Thanks, > Zorro > > > > > --D > > > > > Thanks, > > > Zorro > > > > > > > > > > > > > > --D > > > > > > > > > Thanks, > > > > > Zorro > > > > > > > > > > [1] > > > > > --- /dev/fd/63 2023-09-16 20:33:38.373115157 -0400 > > > > > +++ xfs/270.out.bad 2023-09-16 20:33:37.959115275 -0400 > > > > > @@ -1,6 +1,9 @@ > > > > > QA output created by 270 > > > > > rw mount test > > > > > ro mount test > > > > > +cat: /mnt/xfstests/scratch/testfile: Input/output error > > > > > rw remount test > > > > > rw mount test > > > > > ro mount test > > > > > +ro mount test failed > > > > > +(see /var/lib/xfstests/results//xfs/270.full for details) > > > > > > > > > > [2] > > > > > [57829.436755] run fstests xfs/270 at 2023-09-16 20:33:31 > > > > > [57832.373519] XFS (vda3): Mounting V5 Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > > > > [57832.398699] XFS (vda3): Ending clean mount > > > > > [57832.551895] XFS (vda3): Unmounting Filesystem b31f9652-417d-4dd2-a5ed-54ee957fbfcb > > > > > [57832.909487] XFS (vda3): Mounting V5 Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > > > > [57832.934603] XFS (vda3): Ending clean mount > > > > > [57832.963287] XFS (vda3): User initiated shutdown received. > > > > > [57832.967959] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > > > > [57832.974742] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > > > [57833.002720] XFS (vda3): Unmounting Filesystem 08403da7-afbe-4bfa-a45c-487e28280acc > > > > > [57833.376473] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > > [57833.403380] XFS (vda3): Ending clean mount > > > > > [57833.443719] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > > [57833.516114] [TTM] Buffer eviction failed > > > > > [57833.519490] qxl 0000:00:02.0: object_init failed for (3149824, 0x00000001) > > > > > [57833.523184] [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO > > > > > [57833.955746] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > > [57833.960383] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > > > [57833.964353] XFS (vda3): Filesystem can only be safely mounted read only. > > > > > [57833.967997] XFS (vda3): SB validate failed with error -22. > > > > > [57834.039637] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > > [57834.046289] XFS (vda3): Mounting V5 Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > > [57834.075451] XFS (vda3): Ending clean mount > > > > > [57834.082359] XFS (vda3): Corruption detected in superblock read-only compatible features (0x80000000)! > > > > > [57834.086487] XFS (vda3): Metadata corruption detected at xfs_sb_write_verify+0x11d/0x380 [xfs], xfs_sb block 0x0 > > > > > [57834.090716] XFS (vda3): Unmount and run xfs_repair > > > > > [57834.093708] XFS (vda3): First 128 bytes of corrupted metadata buffer: > > > > > [57834.096953] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 3c 00 00 XFSB.........<.. > > > > > [57834.100635] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > > > > [57834.104330] 00000020: 80 1b 25 ca ac c4 42 32 99 55 dc 34 48 68 04 67 ..%...B2.U.4Hh.g > > > > > [57834.108273] 00000030: 00 00 00 00 00 20 00 06 00 00 00 00 00 00 00 80 ..... .......... > > > > > [57834.112176] 00000040: 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 82 ................ > > > > > [57834.116043] 00000050: 00 00 00 01 00 0f 00 00 00 00 00 04 00 00 00 00 ................ > > > > > [57834.119693] 00000060: 00 00 40 00 b4 a5 02 00 02 00 00 08 00 00 00 00 ..@............. > > > > > [57834.123459] 00000070: 00 00 00 00 00 00 00 00 0c 09 09 03 14 00 00 19 ................ > > > > > [57834.127159] XFS (vda3): Corruption of in-memory data (0x8) detected at _xfs_buf_ioapply+0x50b/0x5f0 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. > > > > > [57834.134557] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > > > [57834.253491] XFS (vda3): ro->rw transition prohibited on unknown (0x80000000) ro-compat filesystem > > > > > [57834.281980] XFS (vda3): Unmounting Filesystem 801b25ca-acc4-4232-9955-dc3448680467 > > > > > [57834.655120] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > > > [57834.682299] XFS (vda3): Ending clean mount > > > > > [57834.723067] XFS (vda3): User initiated shutdown received. > > > > > [57834.730954] XFS (vda3): Metadata I/O Error (0x4) detected at xfs_fs_goingdown+0x4b/0x180 [xfs] (fs/xfs/xfs_fsops.c:492). Shutting down filesystem. > > > > > [57834.738710] XFS (vda3): Please unmount the filesystem and rectify the problem(s) > > > > > [57834.768607] XFS (vda3): Unmounting Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > > > [57835.270472] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > > [57835.275655] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > > > [57835.279434] XFS (vda3): Filesystem can only be safely mounted read only. > > > > > [57835.283124] XFS (vda3): SB validate failed with error -22. > > > > > [57835.357336] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > > [57835.369968] XFS (vda3): Mounting V5 Filesystem 29902d21-a146-4095-8747-562850110ff6 > > > > > [57835.398595] XFS (vda3): Starting recovery (logdev: internal) > > > > > [57835.410361] XFS (vda3): Superblock has unknown read-only compatible features (0x80000000) enabled. > > > > > [57835.414593] XFS (vda3): Attempted to mount read-only compatible filesystem read-write. > > > > > [57835.418495] XFS (vda3): Filesystem can only be safely mounted read only. > > > > > [57835.422152] XFS (vda3): metadata I/O error in "xlog_do_recover+0x3e8/0x540 [xfs]" at daddr 0x0 len 1 error 22 > > > > > [57835.426851] XFS: Assertion failed: 0, file: fs/xfs/xfs_log_recover.c, line: 3321 > > > > > [57835.430765] ------------[ cut here ]------------ > > > > > [57835.434014] WARNING: CPU: 2 PID: 3426232 at fs/xfs/xfs_message.c:104 assfail+0x54/0x70 [xfs] > > > > > [57835.438380] Modules linked in: ext2 overlay dm_zero dm_log_writes dm_thin_pool dm_persistent_data dm_bio_prison sd_mod t10_pi sg dm_snapshot dm_bufio ext4 mbcache jbd2 loop dm_flakey dm_mod tls rfkill intel_rapl_msr sunrpc intel_rapl_common intel_uncore_frequency_common isst_if_common nfit snd_hda_codec_generic ledtrig_audio kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_seq snd_seq_device snd_pcm qxl snd_timer drm_ttm_helper ttm pcspkr virtio_balloon snd drm_kms_helper soundcore i2c_piix4 joydev drm fuse xfs libcrc32c ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix virtio_console virtio_net net_failover failover virtio_blk libata serio_raw [last unloaded: scsi_debug] > > > > > [57835.467794] CPU: 2 PID: 3426232 Comm: mount Kdump: loaded Tainted: G W 6.6.0-rc1+ #1 > > > > > [57835.472233] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > > > > > [57835.475946] RIP: 0010:assfail+0x54/0x70 [xfs] > > > > > [57835.479708] Code: c0 48 ba 00 00 00 00 00 fc ff df 48 89 c1 83 e0 07 48 c1 e9 03 0f b6 14 11 38 c2 7f 04 84 d2 75 10 80 3d 6e 3f 1c 00 00 75 15 <0f> 0b c3 cc cc cc cc 48 c7 c7 10 47 99 c0 e8 79 fd 34 df eb e2 0f > > > > > [57835.489526] RSP: 0018:ffffc9000293f918 EFLAGS: 00010246 > > > > > [57835.493381] RAX: 0000000000000000 RBX: ffff88817b73a000 RCX: 1ffffffff81328e2 > > > > > [57835.497512] RDX: 0000000000000004 RSI: ffffc9000293f680 RDI: ffffffffc0afd780 > > > > > [57835.501588] RBP: ffff8881964ea000 R08: 00000000ffffffea R09: fffff52000527ec1 > > > > > [57835.505701] R10: ffffc9000293f60f R11: fffffffffff6c2b0 R12: 00000000ffffffea > > > > > [57835.509789] R13: ffff88817b73a0e0 R14: ffff8881b927ba80 R15: > > > > > > > > > > > > > > > On Tue, Aug 29, 2023 at 04:09:48PM -0700, Darrick J. Wong wrote: > > > > > > Hi all, > > > > > > > > > > > > Dave pointed out some failures in xfs/270 when he upgraded Debian > > > > > > unstable and util-linux started using the new mount apis. Upon further > > > > > > inquiry I noticed that XFS is quite a hot mess when it encounters a > > > > > > filesystem with unrecognized rocompat bits set in the superblock. > > > > > > > > > > > > Whereas we used to allow readonly mounts under these conditions, a > > > > > > change to the sb write verifier several years ago resulted in the > > > > > > filesystem going down immediately because the post-mount log cleaning > > > > > > writes the superblock, which trips the sb write verifier on the > > > > > > unrecognized rocompat bit. I made the observation that the ROCOMPAT > > > > > > features RMAPBT and REFLINK both protect new log intent item types, > > > > > > which means that we actually cannot support recovering the log if we > > > > > > don't recognize all the rocompat bits. > > > > > > > > > > > > Therefore -- fix inode inactivation to work when we're recovering the > > > > > > log, disallow recovery when there's unrecognized rocompat bits, and > > > > > > don't clean the log if doing so would trip the rocompat checks. > > > > > > > > > > > > v2: change direction of series to allow log recovery on ro mounts > > > > > > > > > > > > If you're going to start using this code, I strongly recommend pulling > > > > > > from my git trees, which are linked below. > > > > > > > > > > > > This has been running on the djcloud for months with no problems. Enjoy! > > > > > > Comments and questions are, as always, welcome. > > > > > > > > > > > > --D > > > > > > > > > > > > fstests git tree: > > > > > > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-ro-mounts > > > > > > --- > > > > > > tests/xfs/270 | 80 ++++++++++++++++++++++++++++++++++++++--------------- > > > > > > tests/xfs/270.out | 2 + > > > > > > 2 files changed, 59 insertions(+), 23 deletions(-) > > > > > > > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-09-19 3:06 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-08-29 23:09 [PATCHSET v2 0/2] fstests: fix ro mounting with unknown rocompat features Darrick J. Wong 2023-08-29 23:09 ` [PATCH 1/2] xfs/270: actually test file readability Darrick J. Wong 2023-09-13 16:54 ` Zorro Lang 2023-08-29 23:09 ` [PATCH 2/2] xfs/270: actually test log recovery with unknown rocompat features Darrick J. Wong 2023-09-17 16:44 ` [PATCHSET v2 0/2] fstests: fix ro mounting " Zorro Lang 2023-09-18 15:17 ` Darrick J. Wong 2023-09-18 16:44 ` Zorro Lang 2023-09-18 18:35 ` Darrick J. Wong 2023-09-19 2:57 ` Zorro Lang 2023-09-19 3:06 ` Darrick J. Wong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox