* Stuck OSD phantom
@ 2012-06-04 1:59 Vladimir Bashkirtsev
2012-06-04 4:08 ` Sage Weil
0 siblings, 1 reply; 3+ messages in thread
From: Vladimir Bashkirtsev @ 2012-06-04 1:59 UTC (permalink / raw)
To: ceph-devel
Dear devs,
While playing around with ceph with six OSDs I decided to retire two
OSDs simultaneously (I do triplication so ceph should withstand such
damage) to see how ceph will cope with it. I was doing it in different
ways trying to get ceph off-rails and it looks I have managed it. :)
First of all I have tried to kill OSDs by pulling them off and then
doing ceph osd lost . Performed as expected. However ceph kept record of
former OSDs even so it did not try to use it. Looks correct.
Then I have recreated OSDs and magically they just came back online and
filled up with data again. Again: that's what is expected.
At last I have tried planned removal of OSDs:
ceph osd crush remove 3
ceph osd rm osd.3
Ceph complained that osd is still up. Shutdown OSD, tried again.
Success. Done the same with second OSD. Everything looked fine still.
And then accidentally (and that's perhaps good test) I have rebooted box
running osd.3 and it had ceph osd in rc. So osd.3 started without having
knowledge that it was evicted from cluster. Cluster magically took it
back and osd.3 joined the culster (however it did not got any load as it
was removed from crush). I removed it from rc, shut it down, done ceph
osd crush remove 3 (just to be certain) and ceph osd rm osd.3 (both
succeeded) but now I have osd.3 still counted towards total cluster
capacity, osd dump shows it as non existent, pg dump shows it as it
still member of a cluster:
[root@x ceph]# ceph osd dump
dumped osdmap epoch 14892
epoch 14892
fsid 7719f573-4c48-4852-a27f-51c7a3fe1c1e
created 2012-03-31 04:47:12.130128
modifed 2012-06-04 11:16:57.687645
flags
pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 192
pgp_num 192 last_change 13812 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins pg_num
192 pgp_num 192 last_change 13815 owner 0
pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 192
pgp_num 192 last_change 13817 owner 0
max_osd 6
osd.0 up in weight 1 up_from 14407 up_thru 14890 down_at 14400
last_clean_interval [14383,14399) 172.16.64.200:6801/25023
172.16.64.200:6802/25023 172.16.64.200:6803/25023 exists,up
osd.1 up in weight 1 up_from 14420 up_thru 14890 down_at 14413
last_clean_interval [14388,14412) lost_at 11147 172.16.64.201:6800/5719
172.16.64.201:6801/5719 172.16.64.201:6802/5719 exists,up
2c7ca892-e83c-4158-a3ae-7c4f96f040b0
osd.4 up in weight 1 up_from 14432 up_thru 14890 down_at 14425
last_clean_interval [14393,14424) lost_at 13373 172.16.64.204:6800/17419
172.16.64.204:6802/17419 172.16.64.204:6803/17419 exists,up
19703275-74c3-403b-8647-85cc4f7ad870
osd.5 up in weight 1 up_from 14448 up_thru 14890 down_at 14438
last_clean_interval [14366,14437) 172.16.64.205:6800/7021
172.16.64.205:6801/7021 172.16.64.205:6802/7021 exists,up
699a39ca-3806-4c4f-9cdc-76cbed61b2ab
[root@x ceph]# ceph pg dump
dumped all in format plain
version 2223459
last_osdmap_epoch 14892
last_pg_scan 12769
full_ratio 0.95
nearfull_ratio 0.85
<-snip->
pool 0 21404 0 0 0 40635329394 23681638 23681638
pool 1 114 0 0 0 234237438 4481899 4481899
pool 2 113241 0 2 0 473699447111 27387426 27387426
sum 134759 0 2 0 514569013943 55550963 55550963
osdstat kbused kbavail kb hb in hb out
0 399440780 316714764 744751104 [1,4,5] []
1 400369588 125798956 546603008 [0,4,5] []
3 130380 90124804 94470144 [0,1,4,5] []
4 387384720 132412912 540409856 [0,1,5] []
5 344705816 233764680 600997888 [0,1,4] []
sum 1532031284 898816116 2527232000
Any idea how to get rid of it completely?
Regards,
Vladimir
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Stuck OSD phantom
2012-06-04 1:59 Stuck OSD phantom Vladimir Bashkirtsev
@ 2012-06-04 4:08 ` Sage Weil
2012-06-04 4:21 ` Vladimir Bashkirtsev
0 siblings, 1 reply; 3+ messages in thread
From: Sage Weil @ 2012-06-04 4:08 UTC (permalink / raw)
To: Vladimir Bashkirtsev; +Cc: ceph-devel
Hi Vladimir,
On Mon, 4 Jun 2012, Vladimir Bashkirtsev wrote:
> Dear devs,
>
> While playing around with ceph with six OSDs I decided to retire two OSDs
> simultaneously (I do triplication so ceph should withstand such damage) to see
> how ceph will cope with it. I was doing it in different ways trying to get
> ceph off-rails and it looks I have managed it. :)
>
> First of all I have tried to kill OSDs by pulling them off and then doing ceph
> osd lost . Performed as expected. However ceph kept record of former OSDs even
> so it did not try to use it. Looks correct.
>
> Then I have recreated OSDs and magically they just came back online and filled
> up with data again. Again: that's what is expected.
>
> At last I have tried planned removal of OSDs:
>
> ceph osd crush remove 3
> ceph osd rm osd.3
>
> Ceph complained that osd is still up. Shutdown OSD, tried again. Success.
> Done the same with second OSD. Everything looked fine still.
>
> And then accidentally (and that's perhaps good test) I have rebooted box
> running osd.3 and it had ceph osd in rc. So osd.3 started without having
> knowledge that it was evicted from cluster. Cluster magically took it back and
> osd.3 joined the culster (however it did not got any load as it was removed
> from crush). I removed it from rc, shut it down, done ceph osd crush remove 3
> (just to be certain) and ceph osd rm osd.3 (both succeeded) but now I have
> osd.3 still counted towards total cluster capacity, osd dump shows it as non
> existent, pg dump shows it as it still member of a cluster:
>
> [root@x ceph]# ceph osd dump
> dumped osdmap epoch 14892
> epoch 14892
> fsid 7719f573-4c48-4852-a27f-51c7a3fe1c1e
> created 2012-03-31 04:47:12.130128
> modifed 2012-06-04 11:16:57.687645
> flags
>
> pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 192
> pgp_num 192 last_change 13812 owner 0 crash_replay_interval 45
> pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins pg_num 192
> pgp_num 192 last_change 13815 owner 0
> pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 192
> pgp_num 192 last_change 13817 owner 0
>
> max_osd 6
> osd.0 up in weight 1 up_from 14407 up_thru 14890 down_at 14400
> last_clean_interval [14383,14399) 172.16.64.200:6801/25023
> 172.16.64.200:6802/25023 172.16.64.200:6803/25023 exists,up
> osd.1 up in weight 1 up_from 14420 up_thru 14890 down_at 14413
> last_clean_interval [14388,14412) lost_at 11147 172.16.64.201:6800/5719
> 172.16.64.201:6801/5719 172.16.64.201:6802/5719 exists,up
> 2c7ca892-e83c-4158-a3ae-7c4f96f040b0
> osd.4 up in weight 1 up_from 14432 up_thru 14890 down_at 14425
> last_clean_interval [14393,14424) lost_at 13373 172.16.64.204:6800/17419
> 172.16.64.204:6802/17419 172.16.64.204:6803/17419 exists,up
> 19703275-74c3-403b-8647-85cc4f7ad870
> osd.5 up in weight 1 up_from 14448 up_thru 14890 down_at 14438
> last_clean_interval [14366,14437) 172.16.64.205:6800/7021
> 172.16.64.205:6801/7021 172.16.64.205:6802/7021 exists,up
> 699a39ca-3806-4c4f-9cdc-76cbed61b2ab
>
> [root@x ceph]# ceph pg dump
> dumped all in format plain
> version 2223459
> last_osdmap_epoch 14892
> last_pg_scan 12769
> full_ratio 0.95
> nearfull_ratio 0.85
> <-snip->
> pool 0 21404 0 0 0 40635329394 23681638 23681638
> pool 1 114 0 0 0 234237438 4481899 4481899
> pool 2 113241 0 2 0 473699447111 27387426 27387426
> sum 134759 0 2 0 514569013943 55550963 55550963
> osdstat kbused kbavail kb hb in hb out
> 0 399440780 316714764 744751104 [1,4,5] []
> 1 400369588 125798956 546603008 [0,4,5] []
> 3 130380 90124804 94470144 [0,1,4,5] []
> 4 387384720 132412912 540409856 [0,1,5] []
> 5 344705816 233764680 600997888 [0,1,4] []
> sum 1532031284 898816116 2527232000
>
> Any idea how to get rid of it completely?
That is a bug; testing a fix now. It's harmless, though, aside from
slightly skewing the used/free stats you see from 'ceph pg stat' or 'df'.
As a workaround, you can create the osd, and then mark it OUT (ceph osd
out 3) first, and then delete it (ceph osd rm 3). Or wait for the next
release, and then recreate and re-delete (ceph osd rm 3) it.
Thanks!
sage
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Stuck OSD phantom
2012-06-04 4:08 ` Sage Weil
@ 2012-06-04 4:21 ` Vladimir Bashkirtsev
0 siblings, 0 replies; 3+ messages in thread
From: Vladimir Bashkirtsev @ 2012-06-04 4:21 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
On 04/06/12 13:38, Sage Weil wrote:
> Hi Vladimir,
>
> On Mon, 4 Jun 2012, Vladimir Bashkirtsev wrote:
>> Dear devs,
>>
>> While playing around with ceph with six OSDs I decided to retire two OSDs
>> simultaneously (I do triplication so ceph should withstand such damage) to see
>> how ceph will cope with it. I was doing it in different ways trying to get
>> ceph off-rails and it looks I have managed it. :)
>>
>> First of all I have tried to kill OSDs by pulling them off and then doing ceph
>> osd lost . Performed as expected. However ceph kept record of former OSDs even
>> so it did not try to use it. Looks correct.
>>
>> Then I have recreated OSDs and magically they just came back online and filled
>> up with data again. Again: that's what is expected.
>>
>> At last I have tried planned removal of OSDs:
>>
>> ceph osd crush remove 3
>> ceph osd rm osd.3
>>
>> Ceph complained that osd is still up. Shutdown OSD, tried again. Success.
>> Done the same with second OSD. Everything looked fine still.
>>
>> And then accidentally (and that's perhaps good test) I have rebooted box
>> running osd.3 and it had ceph osd in rc. So osd.3 started without having
>> knowledge that it was evicted from cluster. Cluster magically took it back and
>> osd.3 joined the culster (however it did not got any load as it was removed
>> from crush). I removed it from rc, shut it down, done ceph osd crush remove 3
>> (just to be certain) and ceph osd rm osd.3 (both succeeded) but now I have
>> osd.3 still counted towards total cluster capacity, osd dump shows it as non
>> existent, pg dump shows it as it still member of a cluster:
>>
>> [root@x ceph]# ceph osd dump
>> dumped osdmap epoch 14892
>> epoch 14892
>> fsid 7719f573-4c48-4852-a27f-51c7a3fe1c1e
>> created 2012-03-31 04:47:12.130128
>> modifed 2012-06-04 11:16:57.687645
>> flags
>>
>> pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 192
>> pgp_num 192 last_change 13812 owner 0 crash_replay_interval 45
>> pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins pg_num 192
>> pgp_num 192 last_change 13815 owner 0
>> pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 192
>> pgp_num 192 last_change 13817 owner 0
>>
>> max_osd 6
>> osd.0 up in weight 1 up_from 14407 up_thru 14890 down_at 14400
>> last_clean_interval [14383,14399) 172.16.64.200:6801/25023
>> 172.16.64.200:6802/25023 172.16.64.200:6803/25023 exists,up
>> osd.1 up in weight 1 up_from 14420 up_thru 14890 down_at 14413
>> last_clean_interval [14388,14412) lost_at 11147 172.16.64.201:6800/5719
>> 172.16.64.201:6801/5719 172.16.64.201:6802/5719 exists,up
>> 2c7ca892-e83c-4158-a3ae-7c4f96f040b0
>> osd.4 up in weight 1 up_from 14432 up_thru 14890 down_at 14425
>> last_clean_interval [14393,14424) lost_at 13373 172.16.64.204:6800/17419
>> 172.16.64.204:6802/17419 172.16.64.204:6803/17419 exists,up
>> 19703275-74c3-403b-8647-85cc4f7ad870
>> osd.5 up in weight 1 up_from 14448 up_thru 14890 down_at 14438
>> last_clean_interval [14366,14437) 172.16.64.205:6800/7021
>> 172.16.64.205:6801/7021 172.16.64.205:6802/7021 exists,up
>> 699a39ca-3806-4c4f-9cdc-76cbed61b2ab
>>
>> [root@x ceph]# ceph pg dump
>> dumped all in format plain
>> version 2223459
>> last_osdmap_epoch 14892
>> last_pg_scan 12769
>> full_ratio 0.95
>> nearfull_ratio 0.85
>> <-snip->
>> pool 0 21404 0 0 0 40635329394 23681638 23681638
>> pool 1 114 0 0 0 234237438 4481899 4481899
>> pool 2 113241 0 2 0 473699447111 27387426 27387426
>> sum 134759 0 2 0 514569013943 55550963 55550963
>> osdstat kbused kbavail kb hb in hb out
>> 0 399440780 316714764 744751104 [1,4,5] []
>> 1 400369588 125798956 546603008 [0,4,5] []
>> 3 130380 90124804 94470144 [0,1,4,5] []
>> 4 387384720 132412912 540409856 [0,1,5] []
>> 5 344705816 233764680 600997888 [0,1,4] []
>> sum 1532031284 898816116 2527232000
>>
>> Any idea how to get rid of it completely?
> That is a bug; testing a fix now. It's harmless, though, aside from
> slightly skewing the used/free stats you see from 'ceph pg stat' or 'df'.
>
> As a workaround, you can create the osd, and then mark it OUT (ceph osd
> out 3) first, and then delete it (ceph osd rm 3). Or wait for the next
> release, and then recreate and re-delete (ceph osd rm 3) it.
>
> Thanks!
> sage
Thank you for quick response! Will wait for next version while trying to
get ceph confused in other way. :)
Regards,
Vladimir
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-06-04 4:22 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-06-04 1:59 Stuck OSD phantom Vladimir Bashkirtsev
2012-06-04 4:08 ` Sage Weil
2012-06-04 4:21 ` Vladimir Bashkirtsev
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.