All of lore.kernel.org
 help / color / mirror / Atom feed
From: Eric Sandeen <sandeen@sandeen.net>
To: Piotr Kandziora <piotr.kandziora@open-e.com>
Cc: Artur Piechocki <artur.piechocki@open-e.com>,
	lukasz.wittig@open-e.com, Janusz Bak <jb@open-e.com>,
	xfs@oss.sgi.com
Subject: Re: XFS: I/O Error Detected / 2.6.27.39
Date: Tue, 16 Nov 2010 15:54:06 -0600	[thread overview]
Message-ID: <4CE2FD7E.1040901@sandeen.net> (raw)
In-Reply-To: <4CE282DB.8060200@open-e.com>

On 11/16/10 7:10 AM, Piotr Kandziora wrote:
> Hi,
> 
> Our environment is following:
> - we have 24GB RAM,
> - we are using 3ware controller (and it does not report any errors),
> - we have one big logical volume (20TB) exported via NFS with large amount of small files (about 150k),
> - we are doing periodically backup of this logical volume using rsync to another server.
> - we have kernel 2.6.27.39,
> 
> Unfortunately our system is freezing unexpectedly without reason. We started investigating this problem and noticed that cache memory is slowly increasing.
> 
> We tried to dump this cache memory using:
> /bin/echo "3" > /proc/sys/vm/drop_caches
> 
> In a result, cache was dumped, but in logs we noticed a lot of errors with XFS:
> 
> [kern.warning] kernel: xfs_iunlink_remove: xfs_inotobp()  returned an error 22 on dm-16.  Returning error.

so this got EINVAL:

#define EINVAL          22      /* Invalid argument */

> [kern.notice] kernel: xfs_inactive:\011xfs_ifree() returned an error = 22 on dm-16
> [kern.notice] kernel: xfs_force_shutdown(dm-16,0x1) called from line 1406 of file fs/xfs/xfs_vnodeops.c.  Return address = 0x
> [kern.alert] kernel: Filesystem \"dm-16\": I/O Error Detected.  Shutting down filesystem: dm-16

which shut down the filesystem:

> [kern.alert] kernel: Please umount the filesystem, and rectify the problem(s)

leading to IO errors after that:

#define EIO              5      /* I/O error */

> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16.  Returning error.
> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16.  Returning error.
> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16.  Returning error.
> 
> We are wondering if this is problem connected to hardware or rather this is XFS problem (if yes, was it fixed?).

I'd probably start with xfs_repair (with -n if you want a dry run)
and see if it finds any metadata corruption, first.

-Eric
 
> Best regards
> Piotr K
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

      parent reply	other threads:[~2010-11-16 21:52 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-11-16 13:10 XFS: I/O Error Detected / 2.6.27.39 Piotr Kandziora
2010-11-16 20:44 ` Emmanuel Florac
2010-11-17 10:43   ` Piotr Kandziora
2010-11-17 11:35     ` Emmanuel Florac
2010-11-17 21:17     ` Michael Monnerie
2010-11-16 21:54 ` Eric Sandeen [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4CE2FD7E.1040901@sandeen.net \
    --to=sandeen@sandeen.net \
    --cc=artur.piechocki@open-e.com \
    --cc=jb@open-e.com \
    --cc=lukasz.wittig@open-e.com \
    --cc=piotr.kandziora@open-e.com \
    --cc=xfs@oss.sgi.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.