new fs, xfs_admin new label, metadata corruption detected

new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

Is this expected? File system continues to work OK, sha256sum on the
single file on this fs matches the source, but the metadata corruption
messages are a bit scary so figured I'd ask about it.

Filesystem created with
xfsprogs-4.3.0-1.fc23
kernel-4.4.4-300.fc23.x86_64

mkfs.xfs defaults used on an LVM thinly provisioned volume on a single
spinning disk.

# xfs_info /dev/mapper/VG-testxfs
meta-data=/dev/mapper/VG-testxfs isize=512    agcount=23, agsize=399984 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0
data     =                       bsize=4096   blocks=8960000, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=3124, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Mounted the file system, copied one decently large ISO, deleted it,
unmounted the file system. Upgraded to kernel 4.5.0, and
xfsprogs-4.3.0-3.fc23, reboot, mounted the file system again, copied a
file, unmounted, and then tried to rename its label:

[root@f23s ~]# xfs_admin -L testxfs /dev/mapper/VG-testxfs
Metadata corruption detected at xfs_agf block 0x30d3808/0x1000
xfs_admin: cannot init perag data (-117). Continuing anyway.
writing all SBs
new label = "testxfs"

Kernel messages:
[snip]
[  226.804218] nf_conntrack: automatic helper assignment is deprecated
and it will be removed soon. Use the iptables CT target to attach
helpers instead.
[  370.527700] SGI XFS with ACLs, security attributes, no debug enabled
[  370.546159] XFS (dm-5): Mounting V5 Filesystem
[  370.621943] XFS (dm-5): Ending clean mount
[  438.064901] EXT4-fs (dm-7): mounted filesystem with ordered data
mode. Opts: (null)
[  837.965252] BTRFS: device label testbtr devid 1 transid 3
/dev/mapper/VG-testbtrfs
[  851.690987] BTRFS info (device dm-8): disk space caching is enabled
[  851.691064] BTRFS: has skinny extents
[  851.691100] BTRFS: flagging fs with big metadata feature
[  851.702530] BTRFS: creating UUID tree
[ 1057.799823] XFS (dm-5): Unmounting Filesystem

[root@f23s ~]# xfs_repair /dev/mapper/VG-testxfs
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
Metadata corruption detected at xfs_agf block 0x4322d08/0x1000
Metadata corruption detected at xfs_agf block 0x30d3808/0x1000
fllast 1014 in agf 22 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x4015988/0x1000
Metadata corruption detected at xfs_agf block 0x3d08608/0x1000
Metadata corruption detected at xfs_agf block 0x33e0b88/0x1000
fllast 1014 in agf 16 too large (max = 1014)
fllast 1014 in agf 21 too large (max = 1014)
fllast 1014 in agf 20 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x39fb288/0x1000
fllast 1014 in agf 17 too large (max = 1014)
fllast 1014 in agf 19 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x36edf08/0x1000
fllast 1014 in agf 18 too large (max = 1014)
        - 16:47:27: scanning filesystem freespace - 23 of 23
allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - 16:47:27: scanning agi unlinked lists - 23 of 23 allocation
groups done
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 15
        - agno = 16
        - agno = 1
        - agno = 2
        - agno = 17
        - agno = 3
        - agno = 18
        - agno = 4
        - agno = 19
        - agno = 5
        - agno = 20
        - agno = 6
        - agno = 7
        - agno = 21
        - agno = 8
        - agno = 22
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - 16:47:27: process known inodes and inode discovery - 64 of
64 inodes done
        - process newly discovered inodes...
        - 16:47:27: process newly discovered inodes - 23 of 23
allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 16:47:27: setting up duplicate extent list - 23 of 23
allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - 16:47:27: check for inodes claiming duplicate blocks - 64 of
64 inodes done
Phase 5 - rebuild AG headers and trees...
        - 16:47:27: rebuild AG headers and trees - 23 of 23 allocation
groups done
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

I haven't tried reproducing this.

-- 
Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Eric Sandeen]

On 3/15/16 6:08 PM, Chris Murphy wrote:
> Is this expected? File system continues to work OK, sha256sum on the
> single file on this fs matches the source, but the metadata corruption
> messages are a bit scary so figured I'd ask about it.
> 
> Filesystem created with
> xfsprogs-4.3.0-1.fc23
> kernel-4.4.4-300.fc23.x86_64
> 
> mkfs.xfs defaults used on an LVM thinly provisioned volume on a single
> spinning disk.
> 
> # xfs_info /dev/mapper/VG-testxfs
> meta-data=/dev/mapper/VG-testxfs isize=512    agcount=23, agsize=399984 blks
                                                ^^^^^^^^^^

It seems that you left a growfs step out of it, where does that come in
to the testcase?

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

[-- Attachment #1.1: Type: text/plain, Size: 1229 bytes --]

On Tue, Mar 15, 2016, 6:34 PM Eric Sandeen <sandeen@sandeen.net> wrote:

> On 3/15/16 6:08 PM, Chris Murphy wrote:
> > Is this expected? File system continues to work OK, sha256sum on the
> > single file on this fs matches the source, but the metadata corruption
> > messages are a bit scary so figured I'd ask about it.
> >
> > Filesystem created with
> > xfsprogs-4.3.0-1.fc23
> > kernel-4.4.4-300.fc23.x86_64
> >
> > mkfs.xfs defaults used on an LVM thinly provisioned volume on a single
> > spinning disk.
> >
> > # xfs_info /dev/mapper/VG-testxfs
> > meta-data=/dev/mapper/VG-testxfs isize=512    agcount=23, agsize=399984
> blks
>                                                 ^^^^^^^^^^
>
> It seems that you left a growfs step out of it, where does that come in
> to the testcase?
>


Oops I spaced it out entirely.

Much of this happened in Cockpit, which uses storaged. So no matter what
I'll have to backtrack to get exact reproduce steps. In the meantime:

Create thin pool
Create thin volume
Mkfs
Mount
Copy file
Grow (resize done in cockpit)
Delete file
Umount
Kernel and progs update and reboot
Mount
Copy file
Unmount
Relabel
<error>
Repair
Mount

Something amiss with growfs and lvresize steps?


Chris Murphy

[-- Attachment #1.2: Type: text/html, Size: 1711 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

OK I can consistently reproduce this on the CLI. I end up with totally different

# lvcreate -L 40g VG -n testxfs
  Logical volume "testxfs" created.
# lvs
  LV         VG   Attr       LSize  Pool       Origin Data%  Meta%
Move Log Cpy%Sync Convert
  testbtrfs  VG   Vwi-a-tz-- 39.06g thintastic        0.01
  testext4   VG   Vwi-a-tz-- 39.06g thintastic        3.96
  testxfs    VG   -wi-a----- 40.00g
  thintastic VG   twi-aotz-- 39.06g                   3.97   4.86
# mkfs.xfs /dev/VG/testxfs
meta-data=/dev/VG/testxfs        isize=512    agcount=4, agsize=2621440 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=4096   blocks=10485760, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=5120, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
# mount /dev/VG/testxfs /mnt/0
# cp file.iso /mnt/0
# lvresize -L 80g VG/testxfs
  Size of logical volume VG/testxfs changed from 40.00 GiB (10240
extents) to 80.00 GiB (20480 extents).
  Logical volume testxfs successfully resized.
# xfs_growfs -d /mnt/0
meta-data=/dev/mapper/VG-testxfs isize=512    agcount=4, agsize=2621440 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0
data     =                       bsize=4096   blocks=10485760, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=5120, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 10485760 to 20971520
# df -h
Filesystem              Size  Used Avail Use% Mounted on
devtmpfs                1.9G     0  1.9G   0% /dev
tmpfs                   1.9G     0  1.9G   0% /dev/shm
tmpfs                   1.9G  904K  1.9G   1% /run
tmpfs                   1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               180G   15G  165G   9% /
tmpfs                   1.9G  4.0K  1.9G   1% /tmp
/dev/sda1               180G   15G  165G   9% /boot
/dev/sda6               525M  8.4M  517M   2% /boot/efi
/dev/mapper/brick1      562G  355G  205G  64% /brick1
tmpfs                   388M     0  388M   0% /run/user/1000
/dev/mapper/VG-testxfs   80G  857M   80G   2% /mnt/0
[root@f23s ~]# lvs
  LV         VG   Attr       LSize  Pool       Origin Data%  Meta%
Move Log Cpy%Sync Convert
  testbtrfs  VG   Vwi-a-tz-- 39.06g thintastic        0.01
  testext4   VG   Vwi-a-tz-- 39.06g thintastic        3.96
  testxfs    VG   -wi-ao---- 80.00g
  thintastic VG   twi-aotz-- 39.06g                   3.97   4.86
# xfs_info /dev/VG/testxfs
meta-data=/dev/mapper/VG-testxfs isize=512    agcount=8, agsize=2621440 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0
data     =                       bsize=4096   blocks=20971520, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=5120, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
# xfs_repair -n /dev/VG/testxfs
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
fllast 1014 in agf 7 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x6400008/0x1000
fllast 1014 in agf 5 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x5000008/0x1000
fllast 1014 in agf 4 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x7800008/0x1000
fllast 1014 in agf 6 too large (max = 1014)
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 4
        - agno = 2
        - agno = 7
        - agno = 5
        - agno = 6
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
# mount /dev/VG/testxfs /mnt/0

No mount errors.
[14189.495433] XFS (dm-5): Mounting V5 Filesystem
[14189.710141] XFS (dm-5): Ending clean mount
[14326.856144] XFS (dm-5): Unmounting Filesystem
[14413.691734] XFS (dm-5): Mounting V5 Filesystem
[14413.833598] XFS (dm-5): Ending clean mount



Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

On Wed, Mar 16, 2016 at 8:39 PM, Chris Murphy <lists@colorremedies.com> wrote:
> OK I can consistently reproduce this on the CLI. I end up with totally different

Ignore the incomplete thought.

-- 
Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

On Wed, Mar 16, 2016 at 8:39 PM, Chris Murphy <lists@colorremedies.com> wrote:

> # xfs_repair -n /dev/VG/testxfs

It's amateur hour... yes I did unmount before running xfs_repair or it
fails. And I also did the test on conventional LV, not thinp.

If I do the same test on a thinp LV, I consistently get more errors
(than conventional LV) when I run xfs_repair, probably because there
are 4x the AG's on thinp LV's for some reason.

# xfs_repair -n /dev/VG/testxfs
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
Metadata corruption detected at xfs_agf block 0x72ff488/0x1000
Metadata corruption detected at xfs_agf block 0x90ff188/0x1000Metadata
corruption detected at xfs_agf block 0x8bff208/0x1000

Metadata corruption detected at xfs_agf block 0x86ff288/0x1000
Metadata corruption detected at xfs_agf block 0x81ff308/0x1000
Metadata corruption detected at xfs_agf block 0x77ff408/0x1000
Metadata corruption detected at xfs_agf block 0x7cff388/0x1000
fllast 1014 in agf 23 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x6dff508/0x1000
Metadata corruption detected at xfs_agf block 0x68ff588/0x1000Metadata
corruption detected at xfs_agf block 0x63ff608/0x1000

Metadata corruption detected at xfs_agf block 0x5eff688/0x1000
Metadata corruption detected at xfs_agf block 0x54ff788/0x1000
Metadata corruption detected at xfs_agf block 0x59ff708/0x1000
Metadata corruption detected at xfs_agf block 0x4fff808/0x1000
fllast 1014 in agf 28 too large (max = 1014)
fllast 1014 in agf 27 too large (max = 1014)
fllast 1014 in agf 26 too large (max = 1014)
fllast 1014 in agf 29 too large (max = 1014)
fllast 1014 in agf 25 too large (max = 1014)
fllast 1014 in agf 24 too large (max = 1014)
fllast 1014 in agf 22 too large (max = 1014)
fllast 1014 in agf 21 too large (max = 1014)
fllast 1014 in agf 19 too large (max = 1014)
fllast 1014 in agf 20 too large (max = 1014)
fllast 1014 in agf 17 too large (max = 1014)
fllast 1014 in agf 18 too large (max = 1014)
fllast 1014 in agf 16 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x9fff008/0x1000
Metadata corruption detected at xfs_agf block 0x95ff108/0x1000
fllast 1014 in agf 32 too large (max = 1014)
fllast 1014 in agf 30 too large (max = 1014)
Metadata corruption detected at xfs_agf block 0x9aff088/0x1000
fllast 1014 in agf 31 too large (max = 1014)
        - 20:51:37: scanning filesystem freespace - 33 of 33
allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 20:51:37: scanning agi unlinked lists - 33 of 33 allocation
groups done
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 30
        - agno = 15
        - agno = 1
        - agno = 31
        - agno = 16
        - agno = 2
        - agno = 3
        - agno = 17
        - agno = 32
        - agno = 4
        - agno = 18
        - agno = 19
        - agno = 5
        - agno = 20
        - agno = 6
        - agno = 21
        - agno = 7
        - agno = 22
        - agno = 8
        - agno = 23
        - agno = 9
        - agno = 24
        - agno = 10
        - agno = 25
        - agno = 11
        - agno = 26
        - agno = 12
        - agno = 27
        - agno = 13
        - agno = 28
        - agno = 14
        - agno = 29
        - 20:51:37: process known inodes and inode discovery - 64 of
64 inodes done
        - process newly discovered inodes...
        - 20:51:37: process newly discovered inodes - 33 of 33
allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 20:51:37: setting up duplicate extent list - 33 of 33
allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - agno = 25
        - agno = 26
        - agno = 27
        - agno = 28
        - agno = 29
        - agno = 30
        - agno = 31
        - agno = 32
        - agno = 4
        - agno = 0
        - agno = 5
        - 20:51:37: check for inodes claiming duplicate blocks - 64 of
64 inodes done
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.



-- 
Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

Using the same procedure but doing it with Btrfs and ext4, there are
no errors. 'btrfs check' and 'btrfs scrub' both come up clean, no
complaints. And e2fsck has no complaints for a resized ext4 volume.


Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Zorro Lang]

On Wed, Mar 16, 2016 at 08:39:21PM -0600, Chris Murphy wrote:
> OK I can consistently reproduce this on the CLI. I end up with totally different
> 
> # lvcreate -L 40g VG -n testxfs
>   Logical volume "testxfs" created.
> # lvs
>   LV         VG   Attr       LSize  Pool       Origin Data%  Meta%
> Move Log Cpy%Sync Convert
>   testbtrfs  VG   Vwi-a-tz-- 39.06g thintastic        0.01
>   testext4   VG   Vwi-a-tz-- 39.06g thintastic        3.96
>   testxfs    VG   -wi-a----- 40.00g
>   thintastic VG   twi-aotz-- 39.06g                   3.97   4.86
> # mkfs.xfs /dev/VG/testxfs
> meta-data=/dev/VG/testxfs        isize=512    agcount=4, agsize=2621440 blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=0
> data     =                       bsize=4096   blocks=10485760, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
> log      =internal log           bsize=4096   blocks=5120, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> # mount /dev/VG/testxfs /mnt/0
> # cp file.iso /mnt/0
> # lvresize -L 80g VG/testxfs
>   Size of logical volume VG/testxfs changed from 40.00 GiB (10240
> extents) to 80.00 GiB (20480 extents).
>   Logical volume testxfs successfully resized.
> # xfs_growfs -d /mnt/0
> meta-data=/dev/mapper/VG-testxfs isize=512    agcount=4, agsize=2621440 blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=1        finobt=1 spinodes=0
> data     =                       bsize=4096   blocks=10485760, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
> log      =internal               bsize=4096   blocks=5120, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> data blocks changed from 10485760 to 20971520
> # df -h
> Filesystem              Size  Used Avail Use% Mounted on
> devtmpfs                1.9G     0  1.9G   0% /dev
> tmpfs                   1.9G     0  1.9G   0% /dev/shm
> tmpfs                   1.9G  904K  1.9G   1% /run
> tmpfs                   1.9G     0  1.9G   0% /sys/fs/cgroup
> /dev/sda1               180G   15G  165G   9% /
> tmpfs                   1.9G  4.0K  1.9G   1% /tmp
> /dev/sda1               180G   15G  165G   9% /boot
> /dev/sda6               525M  8.4M  517M   2% /boot/efi
> /dev/mapper/brick1      562G  355G  205G  64% /brick1
> tmpfs                   388M     0  388M   0% /run/user/1000
> /dev/mapper/VG-testxfs   80G  857M   80G   2% /mnt/0
> [root@f23s ~]# lvs
>   LV         VG   Attr       LSize  Pool       Origin Data%  Meta%
> Move Log Cpy%Sync Convert
>   testbtrfs  VG   Vwi-a-tz-- 39.06g thintastic        0.01
>   testext4   VG   Vwi-a-tz-- 39.06g thintastic        3.96
>   testxfs    VG   -wi-ao---- 80.00g
>   thintastic VG   twi-aotz-- 39.06g                   3.97   4.86
> # xfs_info /dev/VG/testxfs
> meta-data=/dev/mapper/VG-testxfs isize=512    agcount=8, agsize=2621440 blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=1        finobt=1 spinodes=0
> data     =                       bsize=4096   blocks=20971520, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
> log      =internal               bsize=4096   blocks=5120, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> # xfs_repair -n /dev/VG/testxfs
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
> Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
> fllast 1014 in agf 7 too large (max = 1014)
> Metadata corruption detected at xfs_agf block 0x6400008/0x1000
> fllast 1014 in agf 5 too large (max = 1014)
> Metadata corruption detected at xfs_agf block 0x5000008/0x1000
> fllast 1014 in agf 4 too large (max = 1014)
> Metadata corruption detected at xfs_agf block 0x7800008/0x1000
> fllast 1014 in agf 6 too large (max = 1014)

Hi,

I can't reproduce this bug by run the same steps with you said on:
linux 4.5.0-rc2
xfsprogs v4.5.0-rc1

Could you reproduce you problem on the newest linux/xfsprogs?

As above results, agf 4, 5, 6 and 7 are the *new* AGs which created by
growfs. So the corruption only on the *new* AGs. Generally the
agf_fllast should < XFS_AGFL_SIZE(mp), but your fllast is equal to
XFS_AGFL_SIZE(mp).

BTW, if you can reproduce this bug on the newest linux/xfsprogs, could
you check when "crc=0, finobt=0" or "crc=1, finobt=0", it still can be
reproduced? If you can't reproduce on the newest, it maybe a bug of fedora.

Thanks,
Zorro

>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 1
>         - agno = 4
>         - agno = 2
>         - agno = 7
>         - agno = 5
>         - agno = 6
>         - agno = 3
> No modify flag set, skipping phase 5
> Phase 6 - check inode connectivity...
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify link counts...
> No modify flag set, skipping filesystem flush and exiting.
> # mount /dev/VG/testxfs /mnt/0
> 
> No mount errors.
> [14189.495433] XFS (dm-5): Mounting V5 Filesystem
> [14189.710141] XFS (dm-5): Ending clean mount
> [14326.856144] XFS (dm-5): Unmounting Filesystem
> [14413.691734] XFS (dm-5): Mounting V5 Filesystem
> [14413.833598] XFS (dm-5): Ending clean mount
> 
> 
> 
> Chris Murphy
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Dave Chinner]

On Wed, Mar 16, 2016 at 08:39:21PM -0600, Chris Murphy wrote:
> OK I can consistently reproduce this on the CLI. I end up with totally different
> 
> # lvcreate -L 40g VG -n testxfs
>   Logical volume "testxfs" created.
....
[grow]
....
> # xfs_repair -n /dev/VG/testxfs
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
> Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
> fllast 1014 in agf 7 too large (max = 1014)
> Metadata corruption detected at xfs_agf block 0x6400008/0x1000
> fllast 1014 in agf 5 too large (max = 1014)
> Metadata corruption detected at xfs_agf block 0x5000008/0x1000
> fllast 1014 in agf 4 too large (max = 1014)
> Metadata corruption detected at xfs_agf block 0x7800008/0x1000
> fllast 1014 in agf 6 too large (max = 1014)
>         - found root inode chunk

That's caused by commit 96f859d ("libxfs: pack the agfl header
structure so XFS_AGFL_SIZE is correct") and growfs setting the
last free list entry to something that a userspace without the
commit  9fccb9f ("libxfs: pack the agfl header structure so
XFS_AGFL_SIZE is correct") fails to validate correctly.

i.e. update userspace to 4.5.0, and the repair noise will go away.

I think we probably need to fix the growfs code to set it's initial
flfirst/fllast values to be 1/0, not 0/EOFL, and the problem will
then go away....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Dave Chinner]

On Fri, Mar 18, 2016 at 07:57:29AM +1100, Dave Chinner wrote:
> On Wed, Mar 16, 2016 at 08:39:21PM -0600, Chris Murphy wrote:
> > OK I can consistently reproduce this on the CLI. I end up with totally different
> > 
> > # lvcreate -L 40g VG -n testxfs
> >   Logical volume "testxfs" created.
> ....
> [grow]
> ....
> > # xfs_repair -n /dev/VG/testxfs
> > Phase 1 - find and verify superblock...
> > Phase 2 - using internal log
> >         - zero log...
> >         - scan filesystem freespace and inode maps...
> > Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
> > fllast 1014 in agf 7 too large (max = 1014)
> > Metadata corruption detected at xfs_agf block 0x6400008/0x1000
> > fllast 1014 in agf 5 too large (max = 1014)
> > Metadata corruption detected at xfs_agf block 0x5000008/0x1000
> > fllast 1014 in agf 4 too large (max = 1014)
> > Metadata corruption detected at xfs_agf block 0x7800008/0x1000
> > fllast 1014 in agf 6 too large (max = 1014)
> >         - found root inode chunk
> 
> That's caused by commit 96f859d ("libxfs: pack the agfl header
> structure so XFS_AGFL_SIZE is correct") and growfs setting the
> last free list entry to something that a userspace without the
> commit  9fccb9f ("libxfs: pack the agfl header structure so
> XFS_AGFL_SIZE is correct") fails to validate correctly.
> 
> i.e. update userspace to 4.5.0, and the repair noise will go away.
> 
> I think we probably need to fix the growfs code to set it's initial
> flfirst/fllast values to be 1/0, not 0/EOFL, and the problem will
> then go away....

Patch to do this below if you want to test it.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

xfs: Don't wrap growfs AGFL indexes

From: Dave Chinner <dchinner@redhat.com>

Commit 96f859d ("libxfs: pack the agfl header structure so
XFS_AGFL_SIZE is correct") allowed the freelist to use the empty
slot at the end of the freelist on 64 bit systems that was not
being used due to sizeof() rounding up the structure size.

This has caused versions of xfs_repair prior to 4.5.0 (which also
has the fix) to report this as a corruption once the filesystem has
been grown. Older kernels can also have problems (seen from a whacky
container/vm management environment) mounting filesystems grown on a
system with a newer kernel than the vm/container it is deployed on.

To avoid this problem, change the initial free list indexes not to
wrap across the end of the AGFL, hence avoiding the initialisation
of agf_fllast to the last index in the AGFL.

cc: <stable@vger.kernel.org> # 4.4-4.5
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 fs/xfs/xfs_fsops.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_fsops.c b/fs/xfs/xfs_fsops.c
index ee3aaa0a..ca0d3eb 100644
--- a/fs/xfs/xfs_fsops.c
+++ b/fs/xfs/xfs_fsops.c
@@ -243,8 +243,8 @@ xfs_growfs_data_private(
 		agf->agf_roots[XFS_BTNUM_CNTi] = cpu_to_be32(XFS_CNT_BLOCK(mp));
 		agf->agf_levels[XFS_BTNUM_BNOi] = cpu_to_be32(1);
 		agf->agf_levels[XFS_BTNUM_CNTi] = cpu_to_be32(1);
-		agf->agf_flfirst = 0;
-		agf->agf_fllast = cpu_to_be32(XFS_AGFL_SIZE(mp) - 1);
+		agf->agf_flfirst = cpu_to_be32(1);
+		agf->agf_fllast = 0;
 		agf->agf_flcount = 0;
 		tmpsize = agsize - XFS_PREALLOC_BLOCKS(mp);
 		agf->agf_freeblks = cpu_to_be32(tmpsize);

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Chris Murphy]

On Thu, Mar 17, 2016 at 2:57 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Wed, Mar 16, 2016 at 08:39:21PM -0600, Chris Murphy wrote:
>> OK I can consistently reproduce this on the CLI. I end up with totally different
>>
>> # lvcreate -L 40g VG -n testxfs
>>   Logical volume "testxfs" created.
> ....
> [grow]
> ....
>> # xfs_repair -n /dev/VG/testxfs
>> Phase 1 - find and verify superblock...
>> Phase 2 - using internal log
>>         - zero log...
>>         - scan filesystem freespace and inode maps...
>> Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
>> fllast 1014 in agf 7 too large (max = 1014)
>> Metadata corruption detected at xfs_agf block 0x6400008/0x1000
>> fllast 1014 in agf 5 too large (max = 1014)
>> Metadata corruption detected at xfs_agf block 0x5000008/0x1000
>> fllast 1014 in agf 4 too large (max = 1014)
>> Metadata corruption detected at xfs_agf block 0x7800008/0x1000
>> fllast 1014 in agf 6 too large (max = 1014)
>>         - found root inode chunk
>
> That's caused by commit 96f859d ("libxfs: pack the agfl header
> structure so XFS_AGFL_SIZE is correct") and growfs setting the
> last free list entry to something that a userspace without the
> commit  9fccb9f ("libxfs: pack the agfl header structure so
> XFS_AGFL_SIZE is correct") fails to validate correctly.
>
> i.e. update userspace to 4.5.0, and the repair noise will go away.

Yep. The problem is not reproducible with xfsprogs 4.5.0.


-- 
Chris Murphy

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Dave Chinner]

On Thu, Mar 17, 2016 at 05:21:51PM -0600, Chris Murphy wrote:
> On Thu, Mar 17, 2016 at 2:57 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Wed, Mar 16, 2016 at 08:39:21PM -0600, Chris Murphy wrote:
> >> OK I can consistently reproduce this on the CLI. I end up with totally different
> >>
> >> # lvcreate -L 40g VG -n testxfs
> >>   Logical volume "testxfs" created.
> > ....
> > [grow]
> > ....
> >> # xfs_repair -n /dev/VG/testxfs
> >> Phase 1 - find and verify superblock...
> >> Phase 2 - using internal log
> >>         - zero log...
> >>         - scan filesystem freespace and inode maps...
> >> Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
> >> fllast 1014 in agf 7 too large (max = 1014)
> >> Metadata corruption detected at xfs_agf block 0x6400008/0x1000
> >> fllast 1014 in agf 5 too large (max = 1014)
> >> Metadata corruption detected at xfs_agf block 0x5000008/0x1000
> >> fllast 1014 in agf 4 too large (max = 1014)
> >> Metadata corruption detected at xfs_agf block 0x7800008/0x1000
> >> fllast 1014 in agf 6 too large (max = 1014)
> >>         - found root inode chunk
> >
> > That's caused by commit 96f859d ("libxfs: pack the agfl header
> > structure so XFS_AGFL_SIZE is correct") and growfs setting the
> > last free list entry to something that a userspace without the
> > commit  9fccb9f ("libxfs: pack the agfl header structure so
> > XFS_AGFL_SIZE is correct") fails to validate correctly.
> >
> > i.e. update userspace to 4.5.0, and the repair noise will go away.
> 
> Yep. The problem is not reproducible with xfsprogs 4.5.0.

Thanks for confirming that, Chris.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: new fs, xfs_admin new label, metadata corruption detected

[Zorro Lang]

On Thu, Mar 17, 2016 at 05:21:51PM -0600, Chris Murphy wrote:
> On Thu, Mar 17, 2016 at 2:57 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Wed, Mar 16, 2016 at 08:39:21PM -0600, Chris Murphy wrote:
> >> OK I can consistently reproduce this on the CLI. I end up with totally different
> >>
> >> # lvcreate -L 40g VG -n testxfs
> >>   Logical volume "testxfs" created.
> > ....
> > [grow]
> > ....
> >> # xfs_repair -n /dev/VG/testxfs
> >> Phase 1 - find and verify superblock...
> >> Phase 2 - using internal log
> >>         - zero log...
> >>         - scan filesystem freespace and inode maps...
> >> Metadata corruption detected at xfs_agf block 0x8c00008/0x1000
> >> fllast 1014 in agf 7 too large (max = 1014)
> >> Metadata corruption detected at xfs_agf block 0x6400008/0x1000
> >> fllast 1014 in agf 5 too large (max = 1014)
> >> Metadata corruption detected at xfs_agf block 0x5000008/0x1000
> >> fllast 1014 in agf 4 too large (max = 1014)
> >> Metadata corruption detected at xfs_agf block 0x7800008/0x1000
> >> fllast 1014 in agf 6 too large (max = 1014)
> >>         - found root inode chunk
> >
> > That's caused by commit 96f859d ("libxfs: pack the agfl header
> > structure so XFS_AGFL_SIZE is correct") and growfs setting the
> > last free list entry to something that a userspace without the
> > commit  9fccb9f ("libxfs: pack the agfl header structure so
> > XFS_AGFL_SIZE is correct") fails to validate correctly.
> >
> > i.e. update userspace to 4.5.0, and the repair noise will go away.
> 
> Yep. The problem is not reproducible with xfsprogs 4.5.0.

HaHa, Dave always knows more XFS change history:)

BTW, this bug don't need LVM resize, just need
---
  mkfs.xfs -d size=$half_size $dev
  mount $dev $mnt
  xfs_growfs -d $mnt
  umount $mnt
  xfs_repair -n $dev
---
to reproduce it.

And I think xfstests xfs/015 can cover this test.

Thanks,
Zorro
> 
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs