possible xfs corruption

possible xfs corruption

[blacknred]

Hi...

I'm stuck with a storage issue on reboot. Initially doubted the storage, but
dmesg throws these errors. Now wondering whether this is a fs issue? Any
thoughts as to whats going on here?


XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at
line 1237 of file
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c. 
Caller 0xf8aa7892
 [<f8aa774e>] xlog_find_tail+0x91b/0xa4a [xfs]
 [<f8aa7892>] xlog_recover+0x15/0x20b [xfs]
 [<f8aa7892>] xlog_recover+0x15/0x20b [xfs]
 [<f8ab9000>] xfs_buf_get_empty+0x2a/0x33 [xfs]
 [<f8aa36e5>] xfs_log_mount+0x467/0x4ab [xfs]
 [<f8ab702d>] kmem_zalloc+0xd/0x34 [xfs]
 [<f8aa9d8a>] xfs_mountfs+0x9f5/0xdfb [xfs]
 [<f8a92287>] xfs_fs_vcmn_err+0x5f/0x83 [xfs]
 [<f8abeea5>] xfs_mountfs_check_barriers+0xb8/0xd0 [xfs]
 [<f8ab02d2>] xfs_mount+0x75e/0x82a [xfs]
 [<f8aafb74>] xfs_mount+0x0/0x82a [xfs]
 [<f8abf57f>] vfs_mount+0x17/0x1a [xfs]
 [<f8abf42b>] xfs_fs_fill_super+0x68/0x1a5 [xfs]
 [<c04ef1a0>] snprintf+0x1c/0x1f
 [<c04a9b23>] disk_name+0x56/0x60
 [<c047b131>] get_sb_bdev+0xc6/0x113
 [<f8abe9d3>] xfs_fs_get_sb+0x12/0x16 [xfs]
 [<f8abf3c3>] xfs_fs_fill_super+0x0/0x1a5 [xfs]
 [<c047abf6>] vfs_kern_mount+0x7d/0xf2
 [<c047ac9d>] do_kern_mount+0x25/0x36
 [<c048e6aa>] do_mount+0x5fb/0x66b
 [<c04c63ab>] avc_has_perm+0x3c/0x46
 [<c04c63ab>] avc_has_perm+0x3c/0x46
 [<c04c7c1e>] selinux_inode_alloc_security+0x50/0x7b
 [<c04c7619>] inode_doinit_with_dentry+0x8a/0x495
 [<c045ad0a>] get_page_from_freelist+0x96/0x370
 [<c045b04d>] __alloc_pages+0x69/0x2cf
 [<c048d597>] copy_mount_options+0x26/0x109
 [<c048e787>] sys_mount+0x6d/0xa5
 [<c0404ead>] sysenter_past_esp+0x56/0x79

Thanks in advance
David

-- 
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30395558.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[Emmanuel Florac]

Le Tue, 7 Dec 2010 03:49:49 -0800 (PST)
blacknred <leo1783@hotmail.co.uk> écrivait:

> I'm stuck with a storage issue on reboot. Initially doubted the
> storage, but dmesg throws these errors. Now wondering whether this is
> a fs issue? Any thoughts as to whats going on here?
> 

It looks like a part of the filesystem is physically missing. What is
the underlying device? Apparently it's a SmartArray in some HP server,
did you had a power failure? 

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[blacknred]

Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the
server.....



Emmanuel Florac wrote:
> 
> Le Tue, 7 Dec 2010 03:49:49 -0800 (PST)
> blacknred <leo1783@hotmail.co.uk> écrivait:
> 
>> I'm stuck with a storage issue on reboot. Initially doubted the
>> storage, but dmesg throws these errors. Now wondering whether this is
>> a fs issue? Any thoughts as to whats going on here?
>> 
> 
> It looks like a part of the filesystem is physically missing. What is
> the underlying device? Apparently it's a SmartArray in some HP server,
> did you had a power failure? 
> 
> -- 
> ------------------------------------------------------------------------
> Emmanuel Florac     |   Direction technique
>                     |   Intellique
>                     |	<eflorac@intellique.com>
>                     |   +33 1 78 94 84 02
> ------------------------------------------------------------------------
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 
> 

-- 
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30397173.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[Emmanuel Florac]

Le Tue, 7 Dec 2010 07:11:04 -0800 (PST)
blacknred <leo1783@hotmail.co.uk> écrivait:

> Yes, It's a Smart Array in HP Proliant Server.
> No power failure, just upgraded the controller firmware and rebooted
> the server.....
> 

Hu oh, that stinks. Nothing in the firmware release notes? Do you have
any other filesystem on this array? 

First you could try to backup the filesystem metadata using

xfs_metadump -o -w /dev/DEVICE outfile.meta

It can be useful later in case things turn bad.
Does it output any errors? IO errors particularly ?

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[blacknred]

It's all xfs..... and didn't see any I/O errors as well...except that dmesg
is flooded with similar traces to one i posted....

Emmanuel Florac wrote:
> 
> Le Tue, 7 Dec 2010 07:11:04 -0800 (PST)
> blacknred <leo1783@hotmail.co.uk> écrivait:
> 
>> Yes, It's a Smart Array in HP Proliant Server.
>> No power failure, just upgraded the controller firmware and rebooted
>> the server.....
>> 
> 
> Hu oh, that stinks. Nothing in the firmware release notes? Do you have
> any other filesystem on this array? 
> 
> First you could try to backup the filesystem metadata using
> 
> xfs_metadump -o -w /dev/DEVICE outfile.meta
> 
> It can be useful later in case things turn bad.
> Does it output any errors? IO errors particularly ?
> 
> -- 
> ------------------------------------------------------------------------
> Emmanuel Florac     |   Direction technique
>                     |   Intellique
>                     |	<eflorac@intellique.com>
>                     |   +33 1 78 94 84 02
> ------------------------------------------------------------------------
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 
> 

-- 
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30398538.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[Stan Hoeppner]

blacknred put forth on 12/7/2010 11:30 AM:
> 
> It's all xfs..... and didn't see any I/O errors as well...except that dmesg
> is flooded with similar traces to one i posted....

Out of the blue, one OP, with two Proliant servers having storage
problems, within hours of one another, the same day?

Long odds, that.

Did you walk a mile, flat footed, on deep shag carpet, while wearing a
wool sweater, in a room with zero humidity, then walk into the DC and
touch each of these servers, or something? ;)

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 614 bytes --]

On Dienstag, 7. Dezember 2010 blacknred wrote:
> Yes, It's a Smart Array in HP Proliant Server.
> No power failure, just upgraded the controller firmware and rebooted
> the server.....

What Server, controller, and from which firmware version did you go to 
which one?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// ****** Radiointerview zum Thema Spam ******
// http://www.it-podcast.at/archiv.html#podcast-100716
// 
// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[Dave Chinner]

On Tue, Dec 07, 2010 at 03:49:49AM -0800, blacknred wrote:
> 
> Hi...
> 
> I'm stuck with a storage issue on reboot. Initially doubted the storage, but
> dmesg throws these errors. Now wondering whether this is a fs issue? Any
> thoughts as to whats going on here?
> 
> 
> XFS: failed to locate log tail
> XFS: log mount/recovery failed: error 117
> XFS: log mount failed
> XFS mounting filesystem cciss/c0d0
> Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at
> line 1237 of file

Which indicates that the head and/or the tail of the log are not
valid. Can you provide the output of:

# xfs_logprint -d /dev/cciss/c0d0

So we can see what the head/tail values are in the log?

> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c. 

CentOS kernel? How old?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: possible xfs corruption

[Eric Sandeen]

On 12/7/10 4:10 PM, Dave Chinner wrote:
> On Tue, Dec 07, 2010 at 03:49:49AM -0800, blacknred wrote:
>>
>> Hi...
>>
>> I'm stuck with a storage issue on reboot. Initially doubted the storage, but
>> dmesg throws these errors. Now wondering whether this is a fs issue? Any
>> thoughts as to whats going on here?
>>
>>
>> XFS: failed to locate log tail
>> XFS: log mount/recovery failed: error 117
>> XFS: log mount failed
>> XFS mounting filesystem cciss/c0d0
>> Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at
>> line 1237 of file
> 
> Which indicates that the head and/or the tail of the log are not
> valid. Can you provide the output of:
> 
> # xfs_logprint -d /dev/cciss/c0d0
> 
> So we can see what the head/tail values are in the log?
> 
>> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c. 
> 
> CentOS kernel? How old?

Assuning it's centos5, there's really no need to be using an xfs kmod there
anymore, the module shipped with the kernel in recent versions of the OS
is really the one you want to use.  That kmod is ancient and unmaintained.

Although I suspect the storage is more likely at fault here.  :)

-Eric


> Cheers,
> 
> Dave.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs