Re: q. about rbd-header

[Oliver Francke <Oliver.Francke@filoo.de>]

Hi Josh,

On 03/14/2012 10:59 PM, Josh Durgin wrote:
> On 03/14/2012 01:49 PM, Oliver Francke wrote:
>> Well,
>>
>> nobody able to sched some light in?
>> Did some math and found out how to fill the size bytes.
>
> Sorry I didn't respond faster.
>
>> But, one question never got answered:
>>      - why is - with busy VMs - frequently the first block affected,
>>        with the result of damaged 
>> grub-loaders/partition-tables/filesystems?
>>        Is this some NULL/zero pointer thingy in case of ceph-failure?
>
> My guess is that this is not the first object affected, but it's where 
> the loss of an object is most easily noticeable - if an object doesn't 
> exist, it's treated as being full of zeros, which might go undetected 
> for a long time if it's e.g. some temp or log file that's not reread 
> and verified.

well, I responded to Sage with some more infos from one of the images 
where the header is missing... Did not want to bother the list ;)

>
>> If you demand some broken images… we have many of them to investigate,
>> unfortunately.
>
> We'd really like to find the root cause of the problem. One 
> possibility is some bad interaction between osds running different 
> versions. This caused one issue with recovery stxShadow saw yesterday, 
> for example (http://tracker.newdream.net/issues/2132). Had you been 
> doing rolling upgrades of osds before these problems appeared? If so, 
> do you know which versions you had running concurrently?
>
> Are your osds often restarting?
>
> What we'd need to diagnose this are osd logs during recovery with:
>
> debug osd = 20
> debug ms = 1
>
> Once you detect the problem, a log from each replica storing the pg 
> the bad/missing object is in should be enough.
>
> And just to make sure, you aren't writing to these rbd images from 
> multiple places, right? This wouldn't cause the missing header 
> objects, but is likely to cause corruption of the image data. This 
> could happen, for example, by rolling an image back to a snapshot 
> while a vm is running on it.

Currently we don't use snapshots. And of course ensure, a VM is running 
once at a time ;-) And we had some "rolling upgrade", but this was 
_after_ trouble/crashes occured.

Oliver.

>
> Josh
>
>> Maybe this sounds a bit harsh, after the 5th night-shift trying to 
>> repair images
>> and keep customers calm, I think this is forgivable.
>>
>> Oliver.
>>
>> Am 14.03.2012 um 16:05 schrieb Oliver Francke:
>>
>>> Hey,
>>>
>>> anybody out there who could explain the structure of a rbd-header? 
>>> After
>>> last crash we have about 10 images with a:
>>>    2012-03-14 15:22:47.998790 7f45a61e3760 librbd: Error reading
>>> header: 2 No such file or directory
>>> error opening image vm-266-disk-1.rbd: 2 No such file or directory
>>> ... error?
>>> I understand the "rb.x.y"-prefix, the 2 ^ 16hex as block-size. But
>>> the size/count encoding is not intuitive ;)
>>>
>>> Besides one file, where I "created" a header and putted it via "rados
>>> put" back into the pool, and got some files
>>> back, many of the other images with lost headers have different sizes.
>>>
>>> We got bad luck again, too many crashed VM's, too much data-loss...
>>>
>>> Comments welcome ;)
>>>
>>> Oliver.
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 

Oliver Francke

filoo GmbH
Moltkestraße 25a
33330 Gütersloh
HRB4355 AG Gütersloh

Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz

Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html