From: Dave Chinner <david@fromorbit.com>
To: Clay Gerrard <clay.gerrard@gmail.com>
Cc: linux-xfs@vger.kernel.org
Subject: Re: ENODATA on list/stat directory
Date: Fri, 24 Jun 2022 11:37:43 +1000 [thread overview]
Message-ID: <20220624013743.GY227878@dread.disaster.area> (raw)
In-Reply-To: <20220623221309.GU227878@dread.disaster.area>
On Fri, Jun 24, 2022 at 08:13:09AM +1000, Dave Chinner wrote:
> On Thu, Jun 23, 2022 at 02:52:22PM -0500, Clay Gerrard wrote:
> > I work on an object storage system, OpenStack Swift, that has always
> > used xfs on the storage nodes. Our system has encountered many
> > various disk failures and occasionally apparent file system corruption
> > over the years, but we've been noticing something lately that might be
> > "new" and I'm considering how to approach the problem. I'm interested
> > to solicit critique on my current thinking/process - particularly from
> > xfs experts.
> >
> > [root@s8k-sjc3-d01-obj-9 ~]# xfs_bmap
> > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53
> > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53:
> > No data available
> > [root@s8k-sjc3-d01-obj-9 ~]# xfs_db
> > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53
> > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53:
> > No data available
>
> ENODATA implies that it's trying to access an xattr that doesn't
> exist.
>
> > fatal error -- couldn't initialize XFS library
> > [root@s8k-sjc3-d01-obj-9 ~]# ls -alhF /srv/node/d21865/quarantined/objects-1/e53
> > ls: cannot access
> > /srv/node/d21865/quarantined/objects-1/e53/f0418758de4baaa402eb301c5bae3e53:
> > No data available
> > total 4.0K
> > drwxr-xr-x 9 swift swift 318 Jun 7 00:57 ./
> > drwxr-xr-x 33 swift swift 4.0K Jun 23 16:10 ../
> > d????????? ? ? ? ? ? f0418758de4baaa402eb301c5bae3e53/
>
> That's the typical ls output when it couldn't stat() an inode. This
> typically occurs when the inode has been corrupted. On XFS, at
> least, this should result in a corruption warning in the kernel log.
>
> Did you check dmesg for errors?
Just to close the circle on the list thread - this is being caused
by failing storage hardware, not filesystem issue.
> [3600194.452706] sd 0:0:3:0: [sdd] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
> [3600194.452715] sd 0:0:3:0: [sdd] tag#5 Sense Key : Medium Error [current] [descriptor]
> [3600194.452719] sd 0:0:3:0: [sdd] tag#5 Add. Sense: Unrecovered read error
> [3600194.452723] sd 0:0:3:0: [sdd] tag#5 CDB: Read(16) 88 00 00 00 00 03 00 1c 76 30 00 00 00 20 00 00
> [3600194.452727] blk_update_request: critical medium error, dev sdd, sector 12886767168
> [3600194.453801] XFS (sdd): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x3001c7630 len 32 error 61
> [3600194.454846] XFS (sdd): xfs_imap_to_bp: xfs_trans_read_buf() returned error -61.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
prev parent reply other threads:[~2022-06-24 1:37 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-23 19:52 ENODATA on list/stat directory Clay Gerrard
2022-06-23 22:13 ` Dave Chinner
2022-06-24 1:37 ` Dave Chinner [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220624013743.GY227878@dread.disaster.area \
--to=david@fromorbit.com \
--cc=clay.gerrard@gmail.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox