Re: [ceph-users] xfs corruption, data disaster!

Re: [ceph-users] xfs corruption, data disaster!

[Ric Wheeler]

On 05/05/2015 04:13 AM, Yujian Peng wrote:
> Emmanuel Florac <eflorac@...> writes:
>
>> Le Mon, 4 May 2015 07:00:32 +0000 (UTC)
>> Yujian Peng <pengyujian5201314 <at> 126.com> écrivait:
>>
>>> I'm encountering a data disaster. I have a ceph cluster with 145 osd.
>>> The data center had a power problem yesterday, and all of the ceph
>>> nodes were down. But now I find that 6 disks(xfs) in 4 nodes have
>>> data corruption. Some disks are unable to mount, and some disks have
>>> IO errors in syslog. mount: Structure needs cleaning
>>> 	xfs_log_forece: error 5 returned
>>> I tried to repair one with xfs_repair -L /dev/sdx1, but the ceph-osd
>>> reported a leveldb error:
>>> 	Error initializing leveldb: Corruption: checksum mismatch
>>> I cannot start the 6 osds and 22 pgs is down.
>>> This is really a tragedy for me. Can you give me some idea to
>>> recovery the xfs? Thanks very much!
>> For XFS problems, ask the XFS ML: xfs <at> oss.sgi.com
>>
>> You didn't give enough details, by far. What version of kernel and
>> distro are you running? If there were errors, please post extensive
>> logs. If you have IO errors on some disks, you probably MUST replace
>> them before going any further.
>>
>> Why did you run xfs_repair -L ? Did you try xfs_repair without options
>> first? Were you running the very very latest version of xfs_repair
>> (3.2.2) ?
>>
> The OS is ubuntu 12.04.5 with kernel 3.13.0
> uname -a
> Linux ceph19 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20
> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 12.04.5 LTS \n \l
> xfs_repair -V
> xfs_repair version 3.1.7
> I've tried xfs_repair without options, but it showed me some errors, so I
> used the -L option.
> Thanks for your reply!
>

Responding quickly to a couple of things:

* xfs_repair -L wipes out the XFS log, not normally a good thing to do

* replacing disks with IO errors is not a great idea if you still need that 
data. You might want to copy the data from that disk to a new disk (same or 
greater size) and then try to repair that new disk.  A lot depends on the type 
of IO error you see - you might have cable issues, HBA issues, or fairly normal 
read issues (which are not worth replacing a disk for).

You should work with your vendor's support team if you have a support contract 
or post the the XFS devel list (copied above) for help.

Good luck!

Ric



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [ceph-users] xfs corruption, data disaster!

[Eric Sandeen]

On 5/11/15 9:47 AM, Ric Wheeler wrote:
> On 05/05/2015 04:13 AM, Yujian Peng wrote:
>> Emmanuel Florac <eflorac@...> writes:
>>
>>> Le Mon, 4 May 2015 07:00:32 +0000 (UTC)
>>> Yujian Peng <pengyujian5201314 <at> 126.com> écrivait:
>>>
>>>> I'm encountering a data disaster. I have a ceph cluster with 145 osd.
>>>> The data center had a power problem yesterday, and all of the ceph
>>>> nodes were down. But now I find that 6 disks(xfs) in 4 nodes have
>>>> data corruption. Some disks are unable to mount, and some disks have
>>>> IO errors in syslog. mount: Structure needs cleaning
>>>>     xfs_log_forece: error 5 returned
>>>> I tried to repair one with xfs_repair -L /dev/sdx1, but the ceph-osd
>>>> reported a leveldb error:
>>>>     Error initializing leveldb: Corruption: checksum mismatch
>>>> I cannot start the 6 osds and 22 pgs is down.
>>>> This is really a tragedy for me. Can you give me some idea to
>>>> recovery the xfs? Thanks very much!
>>> For XFS problems, ask the XFS ML: xfs <at> oss.sgi.com
>>>
>>> You didn't give enough details, by far. What version of kernel and
>>> distro are you running? If there were errors, please post extensive
>>> logs. If you have IO errors on some disks, you probably MUST replace
>>> them before going any further.
>>>
>>> Why did you run xfs_repair -L ? Did you try xfs_repair without options
>>> first? Were you running the very very latest version of xfs_repair
>>> (3.2.2) ?
>>>
>> The OS is ubuntu 12.04.5 with kernel 3.13.0
>> uname -a
>> Linux ceph19 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20
>> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>> cat /etc/issue
>> Ubuntu 12.04.5 LTS \n \l
>> xfs_repair -V
>> xfs_repair version 3.1.7
>> I've tried xfs_repair without options, but it showed me some errors, so I
>> used the -L option.
>> Thanks for your reply!
>>
> 
> Responding quickly to a couple of things:
> 
> * xfs_repair -L wipes out the XFS log, not normally a good thing to do

And if required due to an unreplayable log, often indicates some problem
with the storage system.  For example a volatile write cache not synced as
needed, and lost along with a power loss, leading to a corrupted and
unreplayable XFS log.

> * replacing disks with IO errors is not a great idea if you still
> need that data. You might want to copy the data from that disk to a
> new disk (same or greater size) and then try to repair that new disk.
> A lot depends on the type of IO error you see - you might have cable
> issues, HBA issues, or fairly normal read issues (which are not worth
> replacing a disk for).

Just a note that XFS sometimes starts saying "IO error" when the filesystem
has shut down; this isn't the same as a block-device-level IO error, but you
haven't posted logs or anything, so I'm just guessing here.

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

-Eric

> You should work with your vendor's support team if you have a support
> contract or post the the XFS devel list (copied above) for help.
> 
> Good luck!
> 
> Ric
> 
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs