How to get disk-location from "metadata I/O error"?

How to get disk-location from "metadata I/O error"?

[Vaughan]

Hi All,


I encounter the following output from time to time. Since xfs is built on
our own virtual disk, it's more likely that our vdisk is buggy.
Can anyone tell me how to figure out the on-disk location given the block
number? Does it follow the following formula?
On-disk location = <block number from metadata I/O error> * <bsize of data
section of xfs_info>
However, the result from the above formula suggests the metadata is out of
range of my disk, confused...

Another question is the corrupted metadata buffer always start with 8 bytes
of '8' character. Does xfs use '8' as any magic label?
I'm using centos-7.2, kernel 3.10.0-327.el7.


XFS (md_d27): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120
[xfs], block 0x1e8ffa08
XFS (md_d27): Unmount and run xfs_repair
XFS (md_d27): First 64 bytes of corrupted metadata buffer:
ffff88042fc69000: 38 38 38 38 38 38 38 38 00 00 00 00 00 00 00 00
88888888........
ffff88042fc69010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
................
ffff88042fc69020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
................
ffff88042fc69030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
................
XFS (md_d27): metadata I/O error: block 0x1e8ffa08
("xfs_trans_read_buf_map") error 117 numblks 8
XFS (md_d27): page discard on page ffffea000b34f900, inode 0x23e, offset
362840064.
XFS (md_d27): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120
[xfs], block 0x1e8ffa08
XFS (md_d27): Unmount and run xfs_repair

# xfs_info /mnt/md/27
meta-data=/dev/md_d27            isize=256    agcount=32, agsize=10682336
blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=341834752, imaxpct=5
         =                       sunit=32     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=166911, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

#cat /proc/partition
253     1728 1367342592 md_d27


-Vaughan

Re: How to get disk-location from "metadata I/O error"?

[Carlos Maiolino]

On Mon, Dec 18, 2017 at 07:16:33PM +0800, Vaughan wrote:
> Hi All,
> 
> 
> I encounter the following output from time to time. Since xfs is built on
> our own virtual disk, it's more likely that our vdisk is buggy.
> Can anyone tell me how to figure out the on-disk location given the block
> number? Does it follow the following formula?
> On-disk location = <block number from metadata I/O error> * <bsize of data
> section of xfs_info>
> However, the result from the above formula suggests the metadata is out of
> range of my disk, confused...

No, your formula is not correct. XFS filesystem blocks are not exactly
sequentially arranged as your formula suggests. It depends on the number of
blocks in each Allocation Group, and the Log2 number used to make calculations
regarding disk addresses.

Easier answer: Use xfs_db to convert it:

xfs_db> fsblock 0x1e8ffa08
xfs_db> daddr
current daddr is 2639245888


Complete answer:

 The number of filesystem blocks inside each Allocation Group is always, the one
presented in xfs_info. As in your example:

agsize=10682336

But to make calculations a bit easier, XFS uses a 'rounded up' log2 number of
the number of blocks in the AG.

So, you can have a 'hole' of filesystem blocks between the end of one AG and the
start of the next AG.

According to the information you provided:

agsize=10682336
agcount=32,

XFS will use "24" as a Rounded up Log2 to keep track of AGs blocks:

In your Case, you have 10682336. So, next power of 2 number is "16777216", which
is 2**24.

But the amount of blocks you have in an AG is: 10682336, so you will have
10682336 valid block numbers + a 'hole' of 6094880 blocks on each AG. This is
just a way to make address translations easier in XFS.

So, to get the disk address of that specific block you first need to know:

- Which AG that FS block is
- The offset of that FS block in the AG

To decimal:
0x1e8ffa08 == 512752136

FSBLOCK	 / AG block count rounded up == AG where the block is
FSBLOCK % AG block count roudned up == Offset into the AG

So:
512752136 / 16777216 == 30 == AG where the block you mentioned is in
512752136 % 16777216 == 9435656 == The offset in that AG

Then you can convert the AG block to it's current disk address using:

AG * total fsblocks per AG + Offset into the AG:

30 * 10682336 + 9435656 = 329905736

Then convert to the current disk address:

329905736 << 3 = 2639245888

And again, the disk address you look for is: 2639245888

> 
> Another question is the corrupted metadata buffer always start with 8 bytes
> of '8' character. Does xfs use '8' as any magic label?

No.

You can use xfs_db to dump the content of that specific block and see if there
is anything else there that might give you a clue of what is going on.

Cheers

> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Carlos

Re: How to get disk-location from "metadata I/O error"?

[Dave Chinner]

On Mon, Dec 18, 2017 at 04:41:50PM +0100, Carlos Maiolino wrote:
> On Mon, Dec 18, 2017 at 07:16:33PM +0800, Vaughan wrote:
> > Hi All,
> > 
> > 
> > I encounter the following output from time to time. Since xfs is built on
> > our own virtual disk, it's more likely that our vdisk is buggy.
> > Can anyone tell me how to figure out the on-disk location given the block
> > number? Does it follow the following formula?
> > On-disk location = <block number from metadata I/O error> * <bsize of data
> > section of xfs_info>
> > However, the result from the above formula suggests the metadata is out of
> > range of my disk, confused...
> 
> No, your formula is not correct. XFS filesystem blocks are not exactly
> sequentially arranged as your formula suggests. It depends on the number of
> blocks in each Allocation Group, and the Log2 number used to make calculations
> regarding disk addresses.
> 
> Easier answer: Use xfs_db to convert it:
> 
> xfs_db> fsblock 0x1e8ffa08
> xfs_db> daddr
> current daddr is 2639245888

One little, important detail is wrong here: metadata block numbers
are reported as daddrs, not fsblocks. daddrs are units of basic
blocks and are a linear offset from the start of the device. Basic
blocks are always 512 bytes. i.e. neither the OP or the above is
correct. :/

XFS (md_d27): metadata I/O error: block 0x1e8ffa08 ("xfs_trans_read_buf_map") error 117 numblks 8 

Hence this is an error from a read at daddr 0x1e8ffa08 for 8 basic
blocks (4096 bytes). Looks like something has overwritten the AGF
with crap...

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com