Re: PG stuck stale

[Josh Durgin <josh.durgin@inktank.com>]

On 08/01/2012 11:25 AM, Josh Durgin wrote:
> On 08/01/2012 08:12 AM, Sylvain Munaut wrote:
>> Hi,
>>
>> I'm doing a few tests on ceph (radosgw more precisely).
>>
>> One of the scenario I'm testing is:
>>   - A radogw bucket stored in a rados pool with size=1 (so no
>> replication)
>>   - Complete/Irrecoverable failure of an OSD ( osd.0 )
>>
>> Now obviously in that situation, some of the placement groups will be
>> completely lost and there will be no way to get the data back and I'm
>> OK with that.
>>
>> But my current issue is that after rebuilding a new osd.0 from
>> scratch, the PG that were previously on it and nowhere else are "stuck
>> stale" and I can't figure out how to tell it that it's OK to loose
>> those data but come back to HEALTHY ...
>
> Those pgs shouldn't be stale. How did you rebuild osd.0? Did you just
> redo ceph-osd --mkfs?
>
> The objects should show up as unfound in ceph -s, and then you can
> deal with them as described here:
>
> http://ceph.com/docs/master/ops/manage/failures/osd/#unfound-objects
>
>> I tried doing 'ceph osd lost 0' after I shut it down and before I
>> start it up from scratch again but that didn't change anything.
>
> This is probably not working due to the stale pgs. Stale means no osd
> is reporting anything about them, so they're probably not being updated
> and marked unfound.
>
>> So how can I make the cluster HEALTHY again ?

I didn't notice before, but since you only have 1 copy of the data,
no other osds will tell the new osd.0 that it should have those pgs.
That's why they're stale. You can force osd.0 to notice the pgs it needs
by running:

ceph pg force_create_pg <pgid>

for each pg that's mapped to osd.0 in 'ceph pg dump'.

Once those are all created, and recovery has noticed that those pgs are
all out of date, the mark_unfound_lost command should work.

Josh

>>
>> Cheers,
>>
>>      Sylvain