* XFS Kernel Panics in CentOS
@ 2012-03-30 22:02 Mark Rechler
2012-03-30 22:44 ` Eric Sandeen
0 siblings, 1 reply; 9+ messages in thread
From: Mark Rechler @ 2012-03-30 22:02 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 1580 bytes --]
Hi Everyone,
We've been getting a lot of errors (across several kernels) and eventually
a kernel panic. Any insight into these errors would be much appreciated.
Errors:
Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line 2112 of file
fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
Call Trace:
[<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
[<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
[<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
[<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
[<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
[<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
[<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
[<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
[<ffffffff8000f550>] generic_permission+0x40/0xca
[<ffffffff8000d902>] permission+0x81/0xc8
[<ffffffff8000999d>] __link_path_walk+0x173/0xf42
[<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
[<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
[<ffffffff8001278e>] getname+0x15b/0x1c2
[<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
[<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
[<ffffffff8008c46e>] default_wake_function+0x0/0xe
[<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
[<ffffffff8002a996>] sys_newlstat+0x19/0x31
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
RIP [<ffffffffff8841bfaf>] :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
RSP <ffff81020752dbc8>
CR2: 00000000000002
<0>Kernel panic - not syncing: Fatal exception
Thanks,
Mark
[-- Attachment #1.2: Type: text/html, Size: 1866 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-03-30 22:02 XFS Kernel Panics in CentOS Mark Rechler
@ 2012-03-30 22:44 ` Eric Sandeen
2012-04-02 15:09 ` Mark Rechler
0 siblings, 1 reply; 9+ messages in thread
From: Eric Sandeen @ 2012-03-30 22:44 UTC (permalink / raw)
To: Mark Rechler; +Cc: xfs
On 3/30/12 5:02 PM, Mark Rechler wrote:
> Hi Everyone,
>
> We've been getting a lot of errors (across several kernels) and eventually a kernel panic. Any insight into these errors would be much appreciated.
>
> Errors:
> Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
Saying which CentOS it is would help ;) And, standard disclaimers about how CentOS doesn't come with upstream _or_ distro support, etc etc...
But xfs_da_do_buf(2) indicates on-disk corruption, having encountered a bad magic number when reading from the disk. Have you tried xfs_repair?
-Eric
> Call Trace:
> [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
> [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
> [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
> [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
> [<ffffffff8000f550>] generic_permission+0x40/0xca
> [<ffffffff8000d902>] permission+0x81/0xc8
> [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
> [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
> [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001278e>] getname+0x15b/0x1c2
> [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
> [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
> [<ffffffff8008c46e>] default_wake_function+0x0/0xe
> [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
> [<ffffffff8002a996>] sys_newlstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
> Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
> RIP [<ffffffffff8841bfaf>] :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
> RSP <ffff81020752dbc8>
> CR2: 00000000000002
> <0>Kernel panic - not syncing: Fatal exception
>
> Thanks,
> Mark
>
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-03-30 22:44 ` Eric Sandeen
@ 2012-04-02 15:09 ` Mark Rechler
2012-04-02 18:03 ` Eric Sandeen
0 siblings, 1 reply; 9+ messages in thread
From: Mark Rechler @ 2012-04-02 15:09 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
[-- Attachment #1.1: Type: text/plain, Size: 2833 bytes --]
Hi Eric,
Thank you for the reply. We are running CentOS 5.8, with the
2.6.18-164.10.1.el5.centos.plus kernel as it was mentioned in a bug report
that has similar behavior, but ultimately a different kernel panic (
http://bugs.centos.org/view.php?id=4089). We have tried running xfs_repair
in the past and it has not proved useful. The odd part is that these are
fresh systems (just installed). If it helps, we are also running glusterfs
on these boxes though load does not always correlate to a kernel panic.
Thanks,
Mark
On Fri, Mar 30, 2012 at 6:44 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> On 3/30/12 5:02 PM, Mark Rechler wrote:
> > Hi Everyone,
> >
> > We've been getting a lot of errors (across several kernels) and
> eventually a kernel panic. Any insight into these errors would be much
> appreciated.
> >
> > Errors:
> > Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line 2112 of
> file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
>
> Saying which CentOS it is would help ;) And, standard disclaimers about
> how CentOS doesn't come with upstream _or_ distro support, etc etc...
>
> But xfs_da_do_buf(2) indicates on-disk corruption, having encountered a
> bad magic number when reading from the disk. Have you tried xfs_repair?
>
> -Eric
>
> > Call Trace:
> > [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
> > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
> > [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
> > [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
> > [<ffffffff8000f550>] generic_permission+0x40/0xca
> > [<ffffffff8000d902>] permission+0x81/0xc8
> > [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
> > [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
> > [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
> > [<ffffffff8001278e>] getname+0x15b/0x1c2
> > [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
> > [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
> > [<ffffffff8008c46e>] default_wake_function+0x0/0xe
> > [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
> > [<ffffffff8002a996>] sys_newlstat+0x19/0x31
> > [<ffffffff8005d229>] tracesys+0x71/0xe0
> > [<ffffffff8005d28d>] tracesys+0xd5/0xe0
> >
> > Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
> > RIP [<ffffffffff8841bfaf>] :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
> > RSP <ffff81020752dbc8>
> > CR2: 00000000000002
> > <0>Kernel panic - not syncing: Fatal exception
> >
> > Thanks,
> > Mark
> >
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
>
[-- Attachment #1.2: Type: text/html, Size: 3737 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-04-02 15:09 ` Mark Rechler
@ 2012-04-02 18:03 ` Eric Sandeen
2012-06-29 4:46 ` Changliang Chen
0 siblings, 1 reply; 9+ messages in thread
From: Eric Sandeen @ 2012-04-02 18:03 UTC (permalink / raw)
To: Mark Rechler; +Cc: xfs
On 4/2/12 8:09 AM, Mark Rechler wrote:
> Hi Eric,
>
> Thank you for the reply. We are running CentOS 5.8, with the
> 2.6.18-164.10.1.el5.centos.plus kernel as it was mentioned in a bug
> report that has similar behavior, but ultimately a different kernel
> panic (http://bugs.centos.org/view.php?id=4089). We have tried
> running xfs_repair in the past and it has not proved useful. The odd
> part is that these are fresh systems (just installed). If it helps,
> we are also running glusterfs on these boxes though load does not
> always correlate to a kernel panic.
I can't say for sure what's in that respun "extra" centos kernel,
but I can say this: the error you hit indicates that xfs read a
buffer, and wound up with a metadata buffer which had unrecognized
magic - i.e. it did not look like metadata as expected. Seeing what
looks like corruption, it shut down.
This reminds me a little of
https://bugzilla.redhat.com/show_bug.cgi?id=512552
which I fixed for RHEL customers a while back, where cancelled
readahead in MD was resulting in xfs thinking a buffer was
uptodate, but in fact it was uninitialized, hence it found
garbage and shut down in this way.
Something similar seems to be happening in your case, if xfs_repair
comes up clean; somehow xfs is getting hold of a buffer which
apparently doesn't match what xfs_repair found to be a consistent
filesystem.
So I might suspect something in the storage stack?
Also please be sure you don't have kmod-xfs or xfs-kmod installed
on your centos box, which is a truly ancient and completely unsupported
backport of xfs from long, long ago.
-Eric
> Thanks,
> Mark
>
> On Fri, Mar 30, 2012 at 6:44 PM, Eric Sandeen <sandeen@sandeen.net <mailto:sandeen@sandeen.net>> wrote:
>
> On 3/30/12 5:02 PM, Mark Rechler wrote:
> > Hi Everyone,
> >
> > We've been getting a lot of errors (across several kernels) and eventually a kernel panic. Any insight into these errors would be much appreciated.
> >
> > Errors:
> > Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
>
> Saying which CentOS it is would help ;) And, standard disclaimers about how CentOS doesn't come with upstream _or_ distro support, etc etc...
>
> But xfs_da_do_buf(2) indicates on-disk corruption, having encountered a bad magic number when reading from the disk. Have you tried xfs_repair?
>
> -Eric
>
> > Call Trace:
> > [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
> > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
> > [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
> > [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
> > [<ffffffff8000f550>] generic_permission+0x40/0xca
> > [<ffffffff8000d902>] permission+0x81/0xc8
> > [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
> > [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
> > [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
> > [<ffffffff8001278e>] getname+0x15b/0x1c2
> > [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
> > [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
> > [<ffffffff8008c46e>] default_wake_function+0x0/0xe
> > [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
> > [<ffffffff8002a996>] sys_newlstat+0x19/0x31
> > [<ffffffff8005d229>] tracesys+0x71/0xe0
> > [<ffffffff8005d28d>] tracesys+0xd5/0xe0
> >
> > Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
> > RIP [<ffffffffff8841bfaf>] :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
> > RSP <ffff81020752dbc8>
> > CR2: 00000000000002
> > <0>Kernel panic - not syncing: Fatal exception
> >
> > Thanks,
> > Mark
> >
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com>
> > http://oss.sgi.com/mailman/listinfo/xfs
>
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-04-02 18:03 ` Eric Sandeen
@ 2012-06-29 4:46 ` Changliang Chen
2012-06-29 4:52 ` Eric Sandeen
0 siblings, 1 reply; 9+ messages in thread
From: Changliang Chen @ 2012-06-29 4:46 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs, Mark Rechler
[-- Attachment #1.1: Type: text/plain, Size: 4790 bytes --]
Hi Eric,
Is this issue resolved? We have been getting the same problem, though
we had upgrated the kernel to 2.6.18-308.8.2.el5.
On Tue, Apr 3, 2012 at 2:03 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> On 4/2/12 8:09 AM, Mark Rechler wrote:
> > Hi Eric,
> >
> > Thank you for the reply. We are running CentOS 5.8, with the
> > 2.6.18-164.10.1.el5.centos.plus kernel as it was mentioned in a bug
> > report that has similar behavior, but ultimately a different kernel
> > panic (http://bugs.centos.org/view.php?id=4089). We have tried
> > running xfs_repair in the past and it has not proved useful. The odd
> > part is that these are fresh systems (just installed). If it helps,
> > we are also running glusterfs on these boxes though load does not
> > always correlate to a kernel panic.
>
> I can't say for sure what's in that respun "extra" centos kernel,
> but I can say this: the error you hit indicates that xfs read a
> buffer, and wound up with a metadata buffer which had unrecognized
> magic - i.e. it did not look like metadata as expected. Seeing what
> looks like corruption, it shut down.
>
> This reminds me a little of
> https://bugzilla.redhat.com/show_bug.cgi?id=512552
> which I fixed for RHEL customers a while back, where cancelled
> readahead in MD was resulting in xfs thinking a buffer was
> uptodate, but in fact it was uninitialized, hence it found
> garbage and shut down in this way.
>
> Something similar seems to be happening in your case, if xfs_repair
> comes up clean; somehow xfs is getting hold of a buffer which
> apparently doesn't match what xfs_repair found to be a consistent
> filesystem.
>
> So I might suspect something in the storage stack?
>
> Also please be sure you don't have kmod-xfs or xfs-kmod installed
> on your centos box, which is a truly ancient and completely unsupported
> backport of xfs from long, long ago.
>
> -Eric
>
> > Thanks,
> > Mark
> >
> > On Fri, Mar 30, 2012 at 6:44 PM, Eric Sandeen <sandeen@sandeen.net<mailto:
> sandeen@sandeen.net>> wrote:
> >
> > On 3/30/12 5:02 PM, Mark Rechler wrote:
> > > Hi Everyone,
> > >
> > > We've been getting a lot of errors (across several kernels) and
> eventually a kernel panic. Any insight into these errors would be much
> appreciated.
> > >
> > > Errors:
> > > Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line
> 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
> >
> > Saying which CentOS it is would help ;) And, standard disclaimers
> about how CentOS doesn't come with upstream _or_ distro support, etc etc...
> >
> > But xfs_da_do_buf(2) indicates on-disk corruption, having
> encountered a bad magic number when reading from the disk. Have you tried
> xfs_repair?
> >
> > -Eric
> >
> > > Call Trace:
> > > [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > > [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
> > > [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
> > > [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
> > > [<ffffffff8000f550>] generic_permission+0x40/0xca
> > > [<ffffffff8000d902>] permission+0x81/0xc8
> > > [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
> > > [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
> > > [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
> > > [<ffffffff8001278e>] getname+0x15b/0x1c2
> > > [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
> > > [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
> > > [<ffffffff8008c46e>] default_wake_function+0x0/0xe
> > > [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
> > > [<ffffffff8002a996>] sys_newlstat+0x19/0x31
> > > [<ffffffff8005d229>] tracesys+0x71/0xe0
> > > [<ffffffff8005d28d>] tracesys+0xd5/0xe0
> > >
> > > Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
> > > RIP [<ffffffffff8841bfaf>]
> :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
> > > RSP <ffff81020752dbc8>
> > > CR2: 00000000000002
> > > <0>Kernel panic - not syncing: Fatal exception
> > >
> > > Thanks,
> > > Mark
> > >
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com>
> > > http://oss.sgi.com/mailman/listinfo/xfs
> >
> >
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
--
Regards,
Cocl
ops manager
19lou Operation & Maintenance Dept
[-- Attachment #1.2: Type: text/html, Size: 6774 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-06-29 4:46 ` Changliang Chen
@ 2012-06-29 4:52 ` Eric Sandeen
2012-06-29 4:58 ` Changliang Chen
0 siblings, 1 reply; 9+ messages in thread
From: Eric Sandeen @ 2012-06-29 4:52 UTC (permalink / raw)
To: Changliang Chen; +Cc: xfs@oss.sgi.com, Mark Rechler
[-- Attachment #1.1: Type: text/plain, Size: 5199 bytes --]
On Jun 29, 2012, at 12:46 AM, Changliang Chen <hqucocl@gmail.com> wrote:
> Hi Eric,
>
> Is this issue resolved? We have been getting the same problem, though we had upgrated the kernel to 2.6.18-308.8.2.el5.
>
I do not know; if it were rhel I'd suggest logging a support ticket. I've not seen anything similar on rhel.
Did you make sure there is no xfs kmod rpm installed? What does modinfo xfs say?
> On Tue, Apr 3, 2012 at 2:03 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> On 4/2/12 8:09 AM, Mark Rechler wrote:
> > Hi Eric,
> >
> > Thank you for the reply. We are running CentOS 5.8, with the
> > 2.6.18-164.10.1.el5.centos.plus kernel as it was mentioned in a bug
> > report that has similar behavior, but ultimately a different kernel
> > panic (http://bugs.centos.org/view.php?id=4089). We have tried
> > running xfs_repair in the past and it has not proved useful. The odd
> > part is that these are fresh systems (just installed). If it helps,
> > we are also running glusterfs on these boxes though load does not
> > always correlate to a kernel panic.
>
> I can't say for sure what's in that respun "extra" centos kernel,
> but I can say this: the error you hit indicates that xfs read a
> buffer, and wound up with a metadata buffer which had unrecognized
> magic - i.e. it did not look like metadata as expected. Seeing what
> looks like corruption, it shut down.
>
> This reminds me a little of
> https://bugzilla.redhat.com/show_bug.cgi?id=512552
> which I fixed for RHEL customers a while back, where cancelled
> readahead in MD was resulting in xfs thinking a buffer was
> uptodate, but in fact it was uninitialized, hence it found
> garbage and shut down in this way.
>
> Something similar seems to be happening in your case, if xfs_repair
> comes up clean; somehow xfs is getting hold of a buffer which
> apparently doesn't match what xfs_repair found to be a consistent
> filesystem.
>
> So I might suspect something in the storage stack?
>
> Also please be sure you don't have kmod-xfs or xfs-kmod installed
> on your centos box, which is a truly ancient and completely unsupported
> backport of xfs from long, long ago.
>
> -Eric
>
> > Thanks,
> > Mark
> >
> > On Fri, Mar 30, 2012 at 6:44 PM, Eric Sandeen <sandeen@sandeen.net <mailto:sandeen@sandeen.net>> wrote:
> >
> > On 3/30/12 5:02 PM, Mark Rechler wrote:
> > > Hi Everyone,
> > >
> > > We've been getting a lot of errors (across several kernels) and eventually a kernel panic. Any insight into these errors would be much appreciated.
> > >
> > > Errors:
> > > Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
> >
> > Saying which CentOS it is would help ;) And, standard disclaimers about how CentOS doesn't come with upstream _or_ distro support, etc etc...
> >
> > But xfs_da_do_buf(2) indicates on-disk corruption, having encountered a bad magic number when reading from the disk. Have you tried xfs_repair?
> >
> > -Eric
> >
> > > Call Trace:
> > > [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
> > > [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
> > > [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
> > > [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
> > > [<ffffffff8000f550>] generic_permission+0x40/0xca
> > > [<ffffffff8000d902>] permission+0x81/0xc8
> > > [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
> > > [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
> > > [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
> > > [<ffffffff8001278e>] getname+0x15b/0x1c2
> > > [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
> > > [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
> > > [<ffffffff8008c46e>] default_wake_function+0x0/0xe
> > > [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
> > > [<ffffffff8002a996>] sys_newlstat+0x19/0x31
> > > [<ffffffff8005d229>] tracesys+0x71/0xe0
> > > [<ffffffff8005d28d>] tracesys+0xd5/0xe0
> > >
> > > Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
> > > RIP [<ffffffffff8841bfaf>] :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
> > > RSP <ffff81020752dbc8>
> > > CR2: 00000000000002
> > > <0>Kernel panic - not syncing: Fatal exception
> > >
> > > Thanks,
> > > Mark
> > >
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com>
> > > http://oss.sgi.com/mailman/listinfo/xfs
> >
> >
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
>
>
> --
>
> Regards,
>
> Cocl
> ops manager
> 19lou Operation & Maintenance Dept
[-- Attachment #1.2: Type: text/html, Size: 7811 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-06-29 4:52 ` Eric Sandeen
@ 2012-06-29 4:58 ` Changliang Chen
2012-06-29 15:04 ` Mark Rechler
0 siblings, 1 reply; 9+ messages in thread
From: Changliang Chen @ 2012-06-29 4:58 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs@oss.sgi.com, Mark Rechler
[-- Attachment #1.1: Type: text/plain, Size: 6083 bytes --]
Hi，
We sure that we haven't installed the xfs kmod,and the modinfo are:
# modinfo xfs
filename: /lib/modules/2.6.18-308.8.2.el5/kernel/fs/xfs/xfs.ko
license: GPL
description: SGI XFS with ACLs, security attributes, large block/inode
numbers, no debug enabled
author: Silicon Graphics, Inc.
srcversion: D37A003AFEE1A42BDD4DD56
depends:
vermagic: 2.6.18-308.8.2.el5 SMP mod_unload gcc-4.1
module_sig:
883f3504fd752a1a91bf303215fc9511247a309f792a2c9d45673dbc457399198719262a50135f0a083e666c424dff9de84f1f5eff01e607decb4921e
On Fri, Jun 29, 2012 at 12:52 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> On Jun 29, 2012, at 12:46 AM, Changliang Chen <hqucocl@gmail.com> wrote:
>
> Hi Eric,
>
> Is this issue resolved? We have been getting the same problem, though
> we had upgrated the kernel to 2.6.18-308.8.2.el5.
>
> I do not know; if it were rhel I'd suggest logging a support ticket. I've
> not seen anything similar on rhel.
>
> Did you make sure there is no xfs kmod rpm installed? What does modinfo
> xfs say?
>
> On Tue, Apr 3, 2012 at 2:03 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
>
>> On 4/2/12 8:09 AM, Mark Rechler wrote:
>> > Hi Eric,
>> >
>> > Thank you for the reply. We are running CentOS 5.8, with the
>> > 2.6.18-164.10.1.el5.centos.plus kernel as it was mentioned in a bug
>> > report that has similar behavior, but ultimately a different kernel
>> > panic (http://bugs.centos.org/view.php?id=4089). We have tried
>> > running xfs_repair in the past and it has not proved useful. The odd
>> > part is that these are fresh systems (just installed). If it helps,
>> > we are also running glusterfs on these boxes though load does not
>> > always correlate to a kernel panic.
>>
>> I can't say for sure what's in that respun "extra" centos kernel,
>> but I can say this: the error you hit indicates that xfs read a
>> buffer, and wound up with a metadata buffer which had unrecognized
>> magic - i.e. it did not look like metadata as expected. Seeing what
>> looks like corruption, it shut down.
>>
>> This reminds me a little of
>> https://bugzilla.redhat.com/show_bug.cgi?id=512552
>> which I fixed for RHEL customers a while back, where cancelled
>> readahead in MD was resulting in xfs thinking a buffer was
>> uptodate, but in fact it was uninitialized, hence it found
>> garbage and shut down in this way.
>>
>> Something similar seems to be happening in your case, if xfs_repair
>> comes up clean; somehow xfs is getting hold of a buffer which
>> apparently doesn't match what xfs_repair found to be a consistent
>> filesystem.
>>
>> So I might suspect something in the storage stack?
>>
>> Also please be sure you don't have kmod-xfs or xfs-kmod installed
>> on your centos box, which is a truly ancient and completely unsupported
>> backport of xfs from long, long ago.
>>
>> -Eric
>>
>> > Thanks,
>> > Mark
>> >
>> > On Fri, Mar 30, 2012 at 6:44 PM, Eric Sandeen <sandeen@sandeen.net<mailto:
>> sandeen@sandeen.net>> wrote:
>> >
>> > On 3/30/12 5:02 PM, Mark Rechler wrote:
>> > > Hi Everyone,
>> > >
>> > > We've been getting a lot of errors (across several kernels) and
>> eventually a kernel panic. Any insight into these errors would be much
>> appreciated.
>> > >
>> > > Errors:
>> > > Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line
>> 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
>> >
>> > Saying which CentOS it is would help ;) And, standard disclaimers
>> about how CentOS doesn't come with upstream _or_ distro support, etc etc...
>> >
>> > But xfs_da_do_buf(2) indicates on-disk corruption, having
>> encountered a bad magic number when reading from the disk. Have you tried
>> xfs_repair?
>> >
>> > -Eric
>> >
>> > > Call Trace:
>> > > [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
>> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
>> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
>> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
>> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
>> > > [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
>> > > [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
>> > > [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
>> > > [<ffffffff8000f550>] generic_permission+0x40/0xca
>> > > [<ffffffff8000d902>] permission+0x81/0xc8
>> > > [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
>> > > [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
>> > > [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
>> > > [<ffffffff8001278e>] getname+0x15b/0x1c2
>> > > [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
>> > > [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
>> > > [<ffffffff8008c46e>] default_wake_function+0x0/0xe
>> > > [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
>> > > [<ffffffff8002a996>] sys_newlstat+0x19/0x31
>> > > [<ffffffff8005d229>] tracesys+0x71/0xe0
>> > > [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>> > >
>> > > Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
>> > > RIP [<ffffffffff8841bfaf>]
>> :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
>> > > RSP <ffff81020752dbc8>
>> > > CR2: 00000000000002
>> > > <0>Kernel panic - not syncing: Fatal exception
>> > >
>> > > Thanks,
>> > > Mark
>> > >
>> > >
>> > > _______________________________________________
>> > > xfs mailing list
>> > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com>
>> > > http://oss.sgi.com/mailman/listinfo/xfs
>> >
>> >
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>>
>
>
>
> --
>
> Regards,
>
> Cocl
> ops manager
> 19lou Operation & Maintenance Dept
>
>
--
Regards,
Cocl
ops manager
19lou Operation & Maintenance Dept
[-- Attachment #1.2: Type: text/html, Size: 8709 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-06-29 4:58 ` Changliang Chen
@ 2012-06-29 15:04 ` Mark Rechler
2012-06-29 21:38 ` Stefan Ring
0 siblings, 1 reply; 9+ messages in thread
From: Mark Rechler @ 2012-06-29 15:04 UTC (permalink / raw)
To: Changliang Chen; +Cc: Eric Sandeen, xfs@oss.sgi.com
[-- Attachment #1.1: Type: text/plain, Size: 6725 bytes --]
Hi Everyone,
It turned out in my case to be related to:
http://oss.sgi.com/bugzilla/show_bug.cgi?id=840
Write barriers were not passed when using LVM/XFS/MegaRAID combined. After
upgrading the kernel to 2.6.39 (used packages from
http://elrepo.org/tiki/tiki-index.php) all XFS issues were resolved. The
other solution was not using LVM.
Hope this helps.
Mark
On Fri, Jun 29, 2012 at 12:58 AM, Changliang Chen <hqucocl@gmail.com> wrote:
> Hi，
>
> We sure that we haven't installed the xfs kmod,and the modinfo are:
>
> # modinfo xfs
> filename: /lib/modules/2.6.18-308.8.2.el5/kernel/fs/xfs/xfs.ko
> license: GPL
> description: SGI XFS with ACLs, security attributes, large block/inode
> numbers, no debug enabled
> author: Silicon Graphics, Inc.
> srcversion: D37A003AFEE1A42BDD4DD56
> depends:
> vermagic: 2.6.18-308.8.2.el5 SMP mod_unload gcc-4.1
> module_sig:
> 883f3504fd752a1a91bf303215fc9511247a309f792a2c9d45673dbc457399198719262a50135f0a083e666c424dff9de84f1f5eff01e607decb4921e
>
> On Fri, Jun 29, 2012 at 12:52 PM, Eric Sandeen <sandeen@sandeen.net>wrote:
>
>> On Jun 29, 2012, at 12:46 AM, Changliang Chen <hqucocl@gmail.com> wrote:
>>
>> Hi Eric,
>>
>> Is this issue resolved? We have been getting the same problem, though
>> we had upgrated the kernel to 2.6.18-308.8.2.el5.
>>
>> I do not know; if it were rhel I'd suggest logging a support ticket.
>> I've not seen anything similar on rhel.
>>
>> Did you make sure there is no xfs kmod rpm installed? What does modinfo
>> xfs say?
>>
>> On Tue, Apr 3, 2012 at 2:03 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
>>
>>> On 4/2/12 8:09 AM, Mark Rechler wrote:
>>> > Hi Eric,
>>> >
>>> > Thank you for the reply. We are running CentOS 5.8, with the
>>> > 2.6.18-164.10.1.el5.centos.plus kernel as it was mentioned in a bug
>>> > report that has similar behavior, but ultimately a different kernel
>>> > panic (http://bugs.centos.org/view.php?id=4089). We have tried
>>> > running xfs_repair in the past and it has not proved useful. The odd
>>> > part is that these are fresh systems (just installed). If it helps,
>>> > we are also running glusterfs on these boxes though load does not
>>> > always correlate to a kernel panic.
>>>
>>> I can't say for sure what's in that respun "extra" centos kernel,
>>> but I can say this: the error you hit indicates that xfs read a
>>> buffer, and wound up with a metadata buffer which had unrecognized
>>> magic - i.e. it did not look like metadata as expected. Seeing what
>>> looks like corruption, it shut down.
>>>
>>> This reminds me a little of
>>> https://bugzilla.redhat.com/show_bug.cgi?id=512552
>>> which I fixed for RHEL customers a while back, where cancelled
>>> readahead in MD was resulting in xfs thinking a buffer was
>>> uptodate, but in fact it was uninitialized, hence it found
>>> garbage and shut down in this way.
>>>
>>> Something similar seems to be happening in your case, if xfs_repair
>>> comes up clean; somehow xfs is getting hold of a buffer which
>>> apparently doesn't match what xfs_repair found to be a consistent
>>> filesystem.
>>>
>>> So I might suspect something in the storage stack?
>>>
>>> Also please be sure you don't have kmod-xfs or xfs-kmod installed
>>> on your centos box, which is a truly ancient and completely unsupported
>>> backport of xfs from long, long ago.
>>>
>>> -Eric
>>>
>>> > Thanks,
>>> > Mark
>>> >
>>> > On Fri, Mar 30, 2012 at 6:44 PM, Eric Sandeen <sandeen@sandeen.net<mailto:
>>> sandeen@sandeen.net>> wrote:
>>> >
>>> > On 3/30/12 5:02 PM, Mark Rechler wrote:
>>> > > Hi Everyone,
>>> > >
>>> > > We've been getting a lot of errors (across several kernels) and
>>> eventually a kernel panic. Any insight into these errors would be much
>>> appreciated.
>>> > >
>>> > > Errors:
>>> > > Filesystem "dm-3": XFS internal error xfs_da_do_buf(2) at line
>>> 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff883c1826
>>> >
>>> > Saying which CentOS it is would help ;) And, standard disclaimers
>>> about how CentOS doesn't come with upstream _or_ distro support, etc etc...
>>> >
>>> > But xfs_da_do_buf(2) indicates on-disk corruption, having
>>> encountered a bad magic number when reading from the disk. Have you tried
>>> xfs_repair?
>>> >
>>> > -Eric
>>> >
>>> > > Call Trace:
>>> > > [<ffffffff883c1725>] :xfs:xfs_da_do_buf+0x503/0x5b1
>>> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
>>> > > [<ffffffff883c1826>] :xfs:xfs_da_read_buf+0x16/0x1b
>>> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
>>> > > [<ffffffff883aeb71>] :xfs:xfs_attr_leaf_get+0x2e/0x99
>>> > > [<ffffffff883aec7f>] :xfs:xfs_attr_fetch+0xa3/0xd5
>>> > > [<ffffffff883a7aa8>] :xfs:xfs_acl_iaccess+0x64/0xd4
>>> > > [<ffffffff883f264a>] :xfs:xfs_check_acl+0x1b/0x2b
>>> > > [<ffffffff8000f550>] generic_permission+0x40/0xca
>>> > > [<ffffffff8000d902>] permission+0x81/0xc8
>>> > > [<ffffffff8000999d>] __link_path_walk+0x173/0xf42
>>> > > [<ffffffff8000e9cc>] link_path_walk+0x42/0xb2
>>> > > [<ffffffff8000cc9c>] do_path_lookup+0x275/0x2f1
>>> > > [<ffffffff8001278e>] getname+0x15b/0x1c2
>>> > > [<ffffffff800236f6>] __user_walk_fd+0x37/0x4c
>>> > > [<ffffffff8003f1f6>] vfs_lstat_fd+0x18/0x47
>>> > > [<ffffffff8008c46e>] default_wake_function+0x0/0xe
>>> > > [<ffffffff800efddf>] sys_lgetxattr+0x4e/0x5f
>>> > > [<ffffffff8002a996>] sys_newlstat+0x19/0x31
>>> > > [<ffffffff8005d229>] tracesys+0x71/0xe0
>>> > > [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>>> > >
>>> > > Code: 0f b6 40 02 89 44 24 04 e9 95 00 00 00 44 0f b6 Z3 44 3b 65
>>> > > RIP [<ffffffffff8841bfaf>]
>>> :xfs:xfs_attr_shortform_getvalue+0x24/0xe2
>>> > > RSP <ffff81020752dbc8>
>>> > > CR2: 00000000000002
>>> > > <0>Kernel panic - not syncing: Fatal exception
>>> > >
>>> > > Thanks,
>>> > > Mark
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > xfs mailing list
>>> > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com>
>>> > > http://oss.sgi.com/mailman/listinfo/xfs
>>> >
>>> >
>>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
>>>
>>
>>
>>
>> --
>>
>> Regards,
>>
>> Cocl
>> ops manager
>> 19lou Operation & Maintenance Dept
>>
>>
>
>
> --
>
> Regards,
>
> Cocl
> ops manager
> 19lou Operation & Maintenance Dept
>
[-- Attachment #1.2: Type: text/html, Size: 9582 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: XFS Kernel Panics in CentOS
2012-06-29 15:04 ` Mark Rechler
@ 2012-06-29 21:38 ` Stefan Ring
0 siblings, 0 replies; 9+ messages in thread
From: Stefan Ring @ 2012-06-29 21:38 UTC (permalink / raw)
To: Mark Rechler; +Cc: xfs@oss.sgi.com
> It turned out in my case to be related to:
> http://oss.sgi.com/bugzilla/show_bug.cgi?id=840
>
> Write barriers were not passed when using LVM/XFS/MegaRAID combined. After
> upgrading the kernel to 2.6.39 (used packages from
> http://elrepo.org/tiki/tiki-index.php) all XFS issues were resolved. The
> other solution was not using LVM.
I fail to see how missing write barriers can make a difference to what
the filesystem reads from the block layer. How is this possible? Does
the block layer not take the pending writes into account when handling
read requests?
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-06-29 21:39 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-03-30 22:02 XFS Kernel Panics in CentOS Mark Rechler
2012-03-30 22:44 ` Eric Sandeen
2012-04-02 15:09 ` Mark Rechler
2012-04-02 18:03 ` Eric Sandeen
2012-06-29 4:46 ` Changliang Chen
2012-06-29 4:52 ` Eric Sandeen
2012-06-29 4:58 ` Changliang Chen
2012-06-29 15:04 ` Mark Rechler
2012-06-29 21:38 ` Stefan Ring
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox