Re: domino-style OSD crash

[Yann Dupont <Yann.Dupont@univ-nantes.fr>]

Le 03/07/2012 21:42, Tommi Virtanen a écrit :
> On Tue, Jul 3, 2012 at 1:40 AM, Yann Dupont <Yann.Dupont@univ-nantes.fr> wrote:
>> Upgraded the kernel to 3.5.0-rc4 + some patches, seems btrfs is OK right
>> now.
>>
>> Tried to restart osd with 0.47.3, then next branch, and today with 0.48.
>>
>> 4 of 8 nodes fails with the same message :
>>
>> ceph version 0.48argonaut (commit:c2b20ca74249892c8e5e40c12aa14446a2bf2030)
>>   1: /usr/bin/ceph-osd() [0x701929]
> ...
>>   13: (leveldb::InternalKeyComparator::FindShortestSeparator(std::string*,
>> leveldb::Slice const&) const+0x4d) [0x6e811d]
> That looks like http://tracker.newdream.net/issues/2563 and the best
> we have for that ticket is "looks like you have a corrupted leveldb
> file". Is this reproducible with a freshly mkfs'ed data partition?
Probably not. I have multiple data volumes on each nodes (I was planning 
xfs vs ext4 vs btrfs benchmarks before being ill) and thoses nodes start 
OK with another data partition .

It's very probable that there is corruption somewhere, due to kernel bug 
, probably triggered by btrfs.

Issue 2563 is probably the same.

I'd like to restart those nodes without formatting them, not because the 
data is valuable, but because if the same thing happens in production, a 
method similar to "fsck" the node could be of great value.

I saw the method to check the leveldb. Will try tomorrow without garantees.

In the case I could repair, do you think a crashed FS as it is right now 
is valuable for you, for future reference , as I saw you can't reproduce 
the problem ? I can make an archive (or a btrfs dump ?), but it will be 
quite big.

Cheers,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html