PG stuck stale

PG stuck stale

[Sylvain Munaut]

Hi,

I'm doing a few tests on ceph (radosgw more precisely).

One of the scenario I'm testing is:
 - A radogw bucket stored in a rados pool with size=1 (so no replication)
 - Complete/Irrecoverable failure of an OSD ( osd.0 )

Now obviously in that situation, some of the placement groups will be
completely lost and there will be no way to get the data back and I'm
OK with that.

But my current issue is that after rebuilding a new osd.0 from
scratch, the PG that were previously on it and nowhere else are "stuck
stale" and I can't figure out how to tell it that it's OK to loose
those data but come back to HEALTHY ...

I tried doing 'ceph osd lost 0' after I shut it down and before I
start it up from scratch again but that didn't change anything.

So how can I make the cluster HEALTHY again ?

Cheers,

    Sylvain

Re: PG stuck stale

[Josh Durgin]

On 08/01/2012 08:12 AM, Sylvain Munaut wrote:
> Hi,
>
> I'm doing a few tests on ceph (radosgw more precisely).
>
> One of the scenario I'm testing is:
>   - A radogw bucket stored in a rados pool with size=1 (so no replication)
>   - Complete/Irrecoverable failure of an OSD ( osd.0 )
>
> Now obviously in that situation, some of the placement groups will be
> completely lost and there will be no way to get the data back and I'm
> OK with that.
>
> But my current issue is that after rebuilding a new osd.0 from
> scratch, the PG that were previously on it and nowhere else are "stuck
> stale" and I can't figure out how to tell it that it's OK to loose
> those data but come back to HEALTHY ...

Those pgs shouldn't be stale. How did you rebuild osd.0? Did you just
redo ceph-osd --mkfs?

The objects should show up as unfound in ceph -s, and then you can
deal with them as described here:

http://ceph.com/docs/master/ops/manage/failures/osd/#unfound-objects

> I tried doing 'ceph osd lost 0' after I shut it down and before I
> start it up from scratch again but that didn't change anything.

This is probably not working due to the stale pgs. Stale means no osd
is reporting anything about them, so they're probably not being updated
and marked unfound.

> So how can I make the cluster HEALTHY again ?
>
> Cheers,
>
>      Sylvain

Re: PG stuck stale

[Josh Durgin]

On 08/01/2012 11:25 AM, Josh Durgin wrote:
> On 08/01/2012 08:12 AM, Sylvain Munaut wrote:
>> Hi,
>>
>> I'm doing a few tests on ceph (radosgw more precisely).
>>
>> One of the scenario I'm testing is:
>>   - A radogw bucket stored in a rados pool with size=1 (so no
>> replication)
>>   - Complete/Irrecoverable failure of an OSD ( osd.0 )
>>
>> Now obviously in that situation, some of the placement groups will be
>> completely lost and there will be no way to get the data back and I'm
>> OK with that.
>>
>> But my current issue is that after rebuilding a new osd.0 from
>> scratch, the PG that were previously on it and nowhere else are "stuck
>> stale" and I can't figure out how to tell it that it's OK to loose
>> those data but come back to HEALTHY ...
>
> Those pgs shouldn't be stale. How did you rebuild osd.0? Did you just
> redo ceph-osd --mkfs?
>
> The objects should show up as unfound in ceph -s, and then you can
> deal with them as described here:
>
> http://ceph.com/docs/master/ops/manage/failures/osd/#unfound-objects
>
>> I tried doing 'ceph osd lost 0' after I shut it down and before I
>> start it up from scratch again but that didn't change anything.
>
> This is probably not working due to the stale pgs. Stale means no osd
> is reporting anything about them, so they're probably not being updated
> and marked unfound.
>
>> So how can I make the cluster HEALTHY again ?

I didn't notice before, but since you only have 1 copy of the data,
no other osds will tell the new osd.0 that it should have those pgs.
That's why they're stale. You can force osd.0 to notice the pgs it needs
by running:

ceph pg force_create_pg <pgid>

for each pg that's mapped to osd.0 in 'ceph pg dump'.

Once those are all created, and recovery has noticed that those pgs are
all out of date, the mark_unfound_lost command should work.

Josh

>>
>> Cheers,
>>
>>      Sylvain

Re: PG stuck stale

[Sylvain Munaut]

Hi,

> I didn't notice before, but since you only have 1 copy of the data,
> no other osds will tell the new osd.0 that it should have those pgs.
> That's why they're stale. You can force osd.0 to notice the pgs it needs
> by running:
>
> ceph pg force_create_pg <pgid>
>
> for each pg that's mapped to osd.0 in 'ceph pg dump'.
>
> Once those are all created, and recovery has noticed that those pgs are
> all out of date, the mark_unfound_lost command should work.

Thanks, it worked.

It didn't go through the "unfound" step though. For the pg that were
empty it was good directly and for the pg that had object in them, it
went to unclean/inactive for a few minutes and then they were back as
just empty pg ... and the cluster was HEALTHY again.

Radosgw however doesn't seem to handle the situation too well because
the bucket that had data in this pg still appears in a 'list' but you
can't delete or list and stat it ...

Cheers,

    Sylvain