Re: xfs corrupted

From: Dave Chinner <david@fromorbit.com>
To: Stefanita Rares Dumitrescu <katmai@keptprivate.com>
Cc: xfs@oss.sgi.com
Subject: Re: xfs corrupted
Date: Wed, 16 Oct 2013 07:26:40 +1100	[thread overview]
Message-ID: <20131015202640.GR4446@dastard> (raw)
In-Reply-To: <525D9E3B.5040507@keptprivate.com>

On Tue, Oct 15, 2013 at 09:57:47PM +0200, Stefanita Rares Dumitrescu wrote:
> Since i am using centos 5.9, the version of the xfsprogs seems to be
> old, so i cloned the new one from sgi.
> 
> I have a machine with 4 gb ram, and 4 gb swap, and it's all been
> eaten up by xfs_repair, and slowed down to a crawl.
> 
> the sdc partition is the one being checked. i am all out of memory
> now. 4 gb phys and 4 gb swap all gone.
> 
> http://pastebin.ca/2467064
> 
> posted to pastebin for better formatting.
> 
> i was using:
> 
> [root@kp4 ~]# xfs_repair -o bhash=16384 -o ihash=16384 -o ag_stride=16 \
> > /dev/sdc >& /tmp/repair.log

You don't have enough RAM to run threaded prefetching and parallel
AG processing. You'd do better to turn prefetching off entirely with
"-P" if you are having OOM problems.

> but now i am trying the -m option to see if the memory can be
> limited, so the server doesn't freeze.
> 
> [root@kp4 ~]# xfs_repair -m 3072 -o ag_stride=16 /dev/sdc >& /tmp/repair.log
> 
> nothing in dmesg either.

Give it another 10-20GB of swap, and it should be fine. xfs_repair
usually only thrashes swap when you don't have enough of it and it
keeps trying to free memory, paging in pages that are in swap to
free cached objects from them. Most of the memory references that
repair makes are quite local, so when pages are swapped out they
generally aren't needed again for a while except when cache reclaim
kicks in. Hence if you give it enough swap that it can grow without
bounds, then it should still be quite efficient.

Keep in mind that badly corrupted filesystems require lots more
memory than clean filesystems to check and repair as there is lots
more intermediate state that repair needs to hold in memory about
partially or incompletely referenced objects. Don't be surprised if
the amount of memory needed to repair a badly broken filesystem is
10-100x the amount of RAM needed to run xfs_repair on the same clean
filesystem....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs