2.6.25 released with bug, which leads to XFS crash?

2.6.25 released with bug, which leads to XFS crash?

[Denys Fedoryshchenko]

Hi again

I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs), 
just it is happening even on heavy load after 1-2 days. IMHO such bugs is critical (same as getting kernel panic, and etc), 
cause they are unrecoverable, causing minor filesystem corruption, and only way to fix them - wakeup sysadmin. Worst thing, it is hapenning at night,
when i restart squid, and probably it is doing agressive unlinking stale cache entries. It doesn't do panic, or even oops, but filesystem will be disconnected, 
and squid will remain in loop trying to restart. Sure it is easy to restart it, but maybe it has to be OOPS? so at least i can do 
sysctl -w kernel.panic_on_oops = 1, and FS will be recovered on reboot.

Just want to warn people who is using XFS on loaded servers to keep attention while using 2.6.25, and if you face same bug, report to bugzilla.

-- 
------
Technical Manager
Virtual ISP S.A.L.
Lebanon

Re: 2.6.25 released with bug, which leads to XFS crash?

[markus reichelt]

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]

* Denys Fedoryshchenko <denys@visp.net.lb> wrote:

> Just want to warn people who is using XFS on loaded servers to keep
> attention while using 2.6.25, and if you face same bug, report to
> bugzilla.

Thanks for the heads-up!

-- 
left blank, right bald

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: 2.6.25 released with bug, which leads to XFS crash?

[David Chinner]

On Thu, Apr 17, 2008 at 09:49:36AM +0300, Denys Fedoryshchenko wrote:
> Hi again
> 
> I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it
> is triggerable on different loaded servers with XFS (squid with aufs),
> just it is happening even on heavy load after 1-2 days. IMHO such bugs is
> critical (same as getting kernel panic, and etc),

Well, yes, and we treat shutdown bugs as such. A filesystem shutdown
is effectively a filesystem panic and is indicative of either a
corruption or a bug. The reality is that it takes time to triage
such a problem that only occurs on one workload on one set of
identical machines once every day or two. This does not make the
problem a release blocker, though.

The other side of it is that problems like this in Linux are often
the result of a bug in a lower layer and not XFS itself. Given this
particular problem seems to be memory corruption it could be anything
that is causing it....

> cause they are unrecoverable, causing minor filesystem corruption, and only
> way to fix them - wakeup sysadmin. Worst thing, it is hapenning at night,
> when i restart squid, and probably it is doing agressive unlinking stale
> cache entries. It doesn't do panic, or even oops, but filesystem will be
> disconnected, > and squid will remain in loop trying to restart. Sure it is
> easy to restart it, but maybe it has to be OOPS? so at least i can do
> sysctl -w kernel.panic_on_oops = 1, and FS will be recovered on reboot.

Rather than fearmongering, perhaps you should ask on the XFS list
(xfs@oss.sgi.com) whether anything like this can be done.  Then you
might have learnt about Documentation/filesystems/xfs.txt and
/proc/sys/fs/xfs/panic_mask:

  fs.xfs.panic_mask             (Min: 0  Default: 0  Max: 127)
        Causes certain error conditions to call BUG(). Value is a bitmask;
        AND together the tags which represent errors which should cause panics:

> Just want to warn people who is using XFS on loaded servers to keep
> attention while using 2.6.25, and if you face same bug, report to bugzilla.

Actually, I'd much prefer XFS bug reports to go to xfs@oss.sgi.com
rather than the kernel bugzilla - that way most of the XFS community
will see the bug report and the triage being done and then there's
no need for spamming lkml like this....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: 2.6.25 released with bug, which leads to XFS crash?

[Eric Sandeen]

Denys Fedoryshchenko wrote:
> Hi again
> 
> I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs),

Oh, aufs was not listed in the bug report.  that's an out of tree
module, right?  Is it completely up to date and safe with 2.6.25?

-Eric

Re: 2.6.25 released with bug, which leads to XFS crash?

[Christoph Hellwig]

On Thu, Apr 17, 2008 at 07:39:50PM -0500, Eric Sandeen wrote:
> Denys Fedoryshchenko wrote:
> > Hi again
> > 
> > I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs),
> 
> Oh, aufs was not listed in the bug report.  that's an out of tree
> module, right?  Is it completely up to date and safe with 2.6.25?

And it's a completely crappy one.  I personally would through away every
bug report with that piece of junk involved.

Re: 2.6.25 released with bug, which leads to XFS crash?

[Denys Fedoryshchenko]

AUFS(same as COSS, UFS, NULL) is way of handling with cache storage in squid, and nothing else. It is userspace and not involved deeply with kernel things.
Since long long time i didn't had any single problem with it.
Kernel is fully vanilla.

On Friday 18 April 2008 10:40, Christoph Hellwig wrote:
> On Thu, Apr 17, 2008 at 07:39:50PM -0500, Eric Sandeen wrote:
> > Denys Fedoryshchenko wrote:
> > > Hi again
> > > 
> > > I reported about http://bugzilla.kernel.org/show_bug.cgi?id=10421 , and it is triggerable on different loaded servers with XFS (squid with aufs),
> > 
> > Oh, aufs was not listed in the bug report.  that's an out of tree
> > module, right?  Is it completely up to date and safe with 2.6.25?
> 
> And it's a completely crappy one.  I personally would through away every
> bug report with that piece of junk involved.
> 
> 

-- 
------
Technical Manager
Virtual ISP S.A.L.
Lebanon

Re: 2.6.25 released with bug, which leads to XFS crash?

[Christoph Hellwig]

On Fri, Apr 18, 2008 at 01:32:30PM +0300, Denys Fedoryshchenko wrote:
> AUFS(same as COSS, UFS, NULL) is way of handling with cache storage in squid, and nothing else. It is userspace and not involved deeply with kernel things.
> Since long long time i didn't had any single problem with it.
> Kernel is fully vanilla.

Ah, okay.  There's also a kernel filesystem driver doing unioning that's
also called aufs that I'm referring to.  sorry for the noise.

Re: 2.6.25 released with bug, which leads to XFS crash?

[Eric Sandeen]

Denys Fedoryshchenko wrote:
> AUFS(same as COSS, UFS, NULL) is way of handling with cache storage in squid, and nothing else. It is userspace and not involved deeply with kernel things.
> Since long long time i didn't had any single problem with it.
> Kernel is fully vanilla.

ah, ok.  I had assumed kernel module too, sorry.

-Eric