All of lore.kernel.org
 help / color / mirror / Atom feed
From: Dave Chinner <david@fromorbit.com>
To: L Ox <lox8096@gmail.com>
Cc: xfs@oss.sgi.com
Subject: Re: 88TB filesystem going off-line without warning
Date: Thu, 4 Apr 2013 15:35:15 +1100	[thread overview]
Message-ID: <20130404043515.GC12011@dastard> (raw)
In-Reply-To: <CA+dre-c0K=AWRNRPgNVWANNkMW++FXEJvQzSe=68_JXyJMeNxQ@mail.gmail.com>

On Tue, Apr 02, 2013 at 11:44:15AM -0700, L Ox wrote:
> Hi,
> 
> We have a new Linux/XFS deployment (about a month old) and randomly without
> warning the XFS filesystem will go off-line. We are running Scientific
> Linux release 5.9 with the latest updates.
> 
> # uname -a
> Linux node24 2.6.18-348.3.1.el5 #1 SMP Mon Mar 11 15:43:13 EDT 2013 x86_64
> x86_64 x86_64 GNU/Linux
> 
> # cat /etc/redhat-release
> Scientific Linux release 5.9 (Boron)
> 
> Here are the errors we see in /var/log/messages after the initial off-line
> event:
> 
> -- snip --
> 
> Apr  2 07:50:28 node24 kernel: xfs_iunlink_remove: xfs_inotobp()  returned
> an error 22 on dm-6.  Returning error.
> Apr  2 07:50:28 node24 kernel: xfs_inactive:  xfs_ifree() returned an error
> = 22 on dm-6

#define EINVAL          22      /* Invalid argument */

That tends to imply a corrupt inode number in the unlinked list
chain.

> Here are the messages after I umount/xfs_repair/mount the filesystem:

What did xfs_repair detect/fix?

> # xfs_repair /dev/mapper/vol_d24-root
> Phase 1 - find and verify superblock...
.....
> Phase 6 - check inode connectivity...
>         - resetting contents of realtime bitmap and summary inodes
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> disconnected inode 202102936036, moving to lost+found
> disconnected inode 215350040250, moving to lost+found
> disconnected inode 215350208634, moving to lost+found
> disconnected inode 271016406074, moving to lost+found

Some inodes that had been unlinked from the directory structure
but not freed. They were probably on an unlinked inode list that
couldn't be walked.

> Any ideas?

If the problem is a one off, there isn't anything that can be done.
If you can reproduce it, try to narrow it down to the simplest case
you can...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

      parent reply	other threads:[~2013-04-04  4:35 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-04-02 18:44 88TB filesystem going off-line without warning L Ox
2013-04-03 19:40 ` Emmanuel Florac
2013-04-04  4:35 ` Dave Chinner [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20130404043515.GC12011@dastard \
    --to=david@fromorbit.com \
    --cc=lox8096@gmail.com \
    --cc=xfs@oss.sgi.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.