Re: OOM on quotacheck (again?)

Dave Chinner
2012-10-02T22:14:31Z

On Tue, Oct 02, 2012 at 10:49:27PM +0200, Volker wrote:
> Hi,
>
> >> If its of any interest, i can supply the stack-traces.
> >
> > Yes, it is of interest, can you post everything you found out about
> > the problem? (dmesg, stack traces, repair output, etc).
>
> Everything posted here is from a single server and its chronologically
> top to bottom. Without having checked each and every stacktrace, it
> looked quite similar on the other servers.
>
> http://pastebin.com/PXquE4sM

So you had a hang on 2.6.37 to do with dquot reclaim, you rebooted
the server into what I think is a 3.6 kernel.

Log recovery failed with "bad clientid 0x0", so no superblock
problem. It does tend to indicate that 2.6.37 wrote bad data to the
log, though. If you reboot into 2.6.37, does log recovery run
successfully? i.e. does the failure only occur on 2.6.37 -> 3.6
with a dirty log?

You then ran xfs_repair -P -L, which threw lots of metadata
away and moved lots of stuff to lost+found.

You them mounted the filesystem on the same kernel (has
xfs_trans_read_buf_map() in the trace, hence the 3.6 version), and
it appears to be hung waiting for IO to complete on a dquot buffer.
That tends to indicate that maybe there's a problem with IO
completion somewhere below the XFS layer.

And if there's a problem below XFS w.r.t. IO compeltion, that also
makes me wonder if the log recovery problem isn't also caused by
something below XFS...

What mount options are you using on the 2.6.37 kernel?

> Sidenote:
> The xfs_repair would not finish without supplying -P, otherwise the
> repair hang in phase 6 (might be related to this bug:
> http://oss.sgi.com/archives/xfs-masters/2011-01/msg00009.html)

If you are upgrading your kernel, you should also upgrade your
xfsprogs installation as well.

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com