osd not in tree

osd not in tree

[Drunkard Zhang]

I installed mon x1, mds x1 and osd x11 in one host, then add some osd
from other hosts, But they are not in osd tree, also not usable, how
can I fix this?

The crush command I used:
ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
row=02, rack=05, host=squid87-log13

The osds in down state is planed, but not added yet.


log3 ~ # ceph -s
   health HEALTH_OK
   monmap e1: 1 mons at {log3=10.205.119.2:6789/0}, election epoch 0,
quorum 0 log3
   osdmap e453: 28 osds: 14 up, 14 in
    pgmap v32649: 960 pgs: 960 active+clean; 1058 GB data, 2140 GB
used, 35422 GB / 39123 GB avail
   mdsmap e688: 1/1/1 up {0=aa=up:active}'
log3 ~ # ceph osd tree
dumped osdmap tree epoch 453
# id weight type name up/down reweight
-1 36 pool default
-3 36 rack unknownrack
-2 36 host log3
0 3 osd.0 up 3
1 3 osd.1 up 3
2 3 osd.2 up 3
3 3 osd.3 up 3
4 3 osd.4 up 3
5 3 osd.5 up 3
6 3 osd.6 up 3
7 3 osd.7 up 3
8 3 osd.8 up 3
9 3 osd.9 up 3
10 3 osd.10 up 3

11 0 osd.11 up 3
12 0 osd.12 up 3
13 0 osd.13 up 3
14 0 osd.14 down 0
15 0 osd.15 down 0
16 0 osd.16 down 0
17 0 osd.17 down 0
18 0 osd.18 down 0
19 0 osd.19 down 0
20 0 osd.20 down 0
21 0 osd.21 down 0
22 0 osd.22 down 0
23 0 osd.23 down 0
24 0 osd.24 down 0
25 0 osd.25 down 0
26 0 osd.26 down 0
27 0 osd.27 down 0

log3 ~ # ceph osd dump
dumped osdmap epoch 453
epoch 453
fsid cc239202-2278-40d0-9274-fdae6d4a0f2c
created 2012-11-07 14:08:18.310361
modifed 2012-11-16 15:09:20.677612
flags

pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
320 pgp_num 320 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
pg_num 320 pgp_num 320 last_change 6 owner 0
pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num
320 pgp_num 320 last_change 1 owner 0

max_osd 28
osd.0 up   in  weight 3 up_from 400 up_thru 407 down_at 399
last_clean_interval [371,398) 10.205.119.2:6801/12912
10.205.119.2:6802/12912 10.205.119.2:6803/12912 exists,up
567885be-8edf-4f71-bb2b-410dd973b9e8
osd.1 up   in  weight 3 up_from 401 up_thru 407 down_at 400
last_clean_interval [372,398) 10.205.119.2:6806/13004
10.205.119.2:6809/13004 10.205.119.2:6812/13004 exists,up
617db970-0e2f-4188-93b4-8a825194c359
osd.2 up   in  weight 3 up_from 401 up_thru 407 down_at 400
last_clean_interval [373,398) 10.205.119.2:6818/13333
10.205.119.2:6820/13333 10.205.119.2:6821/13333 exists,up
887e2169-880a-47a9-83c6-11f83f166ed2
osd.3 up   in  weight 3 up_from 402 up_thru 407 down_at 401
last_clean_interval [373,398) 10.205.119.2:6824/13454
10.205.119.2:6827/13454 10.205.119.2:6830/13454 exists,up
237ab216-d405-4e60-a3ca-3ad2c1c2ec75
osd.4 up   in  weight 3 up_from 403 up_thru 407 down_at 402
last_clean_interval [374,398) 10.205.119.2:6833/13550
10.205.119.2:6836/13550 10.205.119.2:6837/13550 exists,up
c3ed8bc8-3393-487a-ace5-ee5d41acceb2
osd.5 up   in  weight 3 up_from 402 up_thru 407 down_at 401
last_clean_interval [374,398) 10.205.119.2:6838/13675
10.205.119.2:6839/13675 10.205.119.2:6840/13675 exists,up
0f6772ea-aab3-4b9b-a33c-44d04b8ba953
osd.6 up   in  weight 3 up_from 403 up_thru 407 down_at 402
last_clean_interval [374,398) 10.205.119.2:6841/13854
10.205.119.2:6842/13854 10.205.119.2:6843/13854 exists,up
2e7593ff-0acf-4f0d-a0a6-506702aed6f6
osd.7 up   in  weight 3 up_from 403 up_thru 407 down_at 402
last_clean_interval [396,398) 10.205.119.2:6844/14007
10.205.119.2:6845/14007 10.205.119.2:6846/14007 exists,up
6f236563-9c8f-4e61-ab96-84c57c9f6f96
osd.8 up   in  weight 3 up_from 404 up_thru 407 down_at 403
last_clean_interval [393,398) 10.205.119.2:6847/14145
10.205.119.2:6848/14145 10.205.119.2:6849/14145 exists,up
0d157aa5-528a-4cbb-ba1b-6c349b1f0d79
osd.9 up   in  weight 3 up_from 407 up_thru 407 down_at 406
last_clean_interval [382,398) 10.205.119.2:6850/14280
10.205.119.2:6851/14280 10.205.119.2:6852/14280 exists,up
a5036100-461a-450b-819e-e38f722ddf93
osd.10 up   in  weight 3 up_from 404 up_thru 407 down_at 403
last_clean_interval [377,398) 10.205.119.2:6814/13093
10.205.119.2:6815/13093 10.205.119.2:6816/13093 exists,up
19a88fc9-9f02-4ce8-a8d5-22b64e62142c
osd.11 up   in  weight 3 up_from 453 up_thru 277 down_at 452
last_clean_interval [343,452) 150.164.100.219:6800/827
150.164.100.219:6805/827 150.164.100.219:6808/827 exists,up
79a80fd3-f76e-4fa7-a623-433f328573f7
osd.12 up   in  weight 3 up_from 445 up_thru 0 down_at 444
last_clean_interval [345,444) 150.164.100.219:6804/1126
150.164.100.219:6809/1126 150.164.100.219:6803/1126 exists,up
5a5ff961-b122-4814-b4e7-35340b7f1bd6
osd.13 up   in  weight 3 up_from 434 up_thru 0 down_at 433
last_clean_interval [347,433) 150.164.100.219:6807/1415
150.164.100.219:6802/1415 150.164.100.219:6806/1415 exists,up
2e6aa5a1-a01f-4943-9efc-2e023b1f7db9
osd.14 down out weight 0 up_from 432 up_thru 0 down_at 433
last_clean_interval [349,431) 150.164.100.219:6810/1703
150.164.100.219:6808/1703 150.164.100.219:6811/1703 autoout,exists
540658ec-7daa-4231-8586-2b86d61fbdba
osd.15 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.16 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.17 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.18 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.19 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.20 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.21 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.22 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.23 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.24 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.25 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.26 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new
osd.27 down out weight 0 up_from 0 up_thru 0 down_at 0
last_clean_interval [0,0) :/0 :/0 :/0 exists,new

The osd on squid87-log13 is not used at all:
squid87-log13 ~ # df | grep osd
/dev/sdb1       2.8T  1.1G  2.7T   1% /ceph/osd.11
/dev/sdc1       2.8T 1009M  2.7T   1% /ceph/osd.12
/dev/sdd1       2.8T 1009M  2.7T   1% /ceph/osd.13
/dev/sde1       2.8T 1009M  2.7T   1% /ceph/osd.14
squid87-log13 ~ # netstat -ntulp | grep osd
tcp        0      0 0.0.0.0:6800            0.0.0.0:*
LISTEN      829/ceph-osd
tcp        0      0 150.164.100.219:6802    0.0.0.0:*
LISTEN      1417/ceph-osd
tcp        0      0 150.164.100.219:6803    0.0.0.0:*
LISTEN      1128/ceph-osd
tcp        0      0 0.0.0.0:6804            0.0.0.0:*
LISTEN      1128/ceph-osd
tcp        0      0 150.164.100.219:6805    0.0.0.0:*
LISTEN      829/ceph-osd
tcp        0      0 150.164.100.219:6806    0.0.0.0:*
LISTEN      1417/ceph-osd
tcp        0      0 0.0.0.0:6807            0.0.0.0:*
LISTEN      1417/ceph-osd
tcp        0      0 150.164.100.219:6808    0.0.0.0:*
LISTEN      829/ceph-osd
tcp        0      0 150.164.100.219:6809    0.0.0.0:*
LISTEN      1128/ceph-osd

Re: osd not in tree

[Josh Durgin]

On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
> from other hosts, But they are not in osd tree, also not usable, how
> can I fix this?
>
> The crush command I used:
> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
> row=02, rack=05, host=squid87-log13

Remove the commas in that command and it'll work. I fixed the docs for
this.

Josh

Re: osd not in tree

[Drunkard Zhang]

2012/11/16 Josh Durgin <josh.durgin@inktank.com>:
> On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
>>
>> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
>> from other hosts, But they are not in osd tree, also not usable, how
>> can I fix this?
>>
>> The crush command I used:
>> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
>> row=02, rack=05, host=squid87-log13
>
>
> Remove the commas in that command and it'll work. I fixed the docs for
> this.
>
> Josh

Got no luck. osd.11.log said nothing useful. Do I have to edit
crushmap manually? If so, how should I define host's 'id' value, I
cannot find docs from ceph.com/docs since yesterday, where's that
gone?

squid87-log13 ~ # ceph osd crush set 11 osd.11 3 pool=data
datacenter=dh-1L room=room1 row=02 rack=05 host=squid87-log13
updated item id 11 name 'osd.11' weight 3 at location
{datacenter=dh-1L,host=squid87-log13,pool=data,rack=05,room=room1,row=02}
to crush map
squid87-log13 ~ # ceph osd tree
dumped osdmap tree epoch 467
# id weight type name up/down reweight
-1 36 pool default
-3 36 rack unknownrack
-2 36 host log3
0 3 osd.0 up 3
1 3 osd.1 up 3
2 3 osd.2 up 3
3 3 osd.3 up 3
4 3 osd.4 up 3
5 3 osd.5 up 3
6 3 osd.6 up 3
7 3 osd.7 up 3
8 3 osd.8 up 3
9 3 osd.9 up 3
10 3 osd.10 up 3

11 0 osd.11 up 3
12 0 osd.12 up 3
13 0 osd.13 up 3
14 0 osd.14 up 3

Re: osd not in tree

[Sage Weil]

On Fri, 16 Nov 2012, Drunkard Zhang wrote:
> 2012/11/16 Josh Durgin <josh.durgin@inktank.com>:
> > On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
> >>
> >> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
> >> from other hosts, But they are not in osd tree, also not usable, how
> >> can I fix this?
> >>
> >> The crush command I used:
> >> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
> >> row=02, rack=05, host=squid87-log13
> >
> >
> > Remove the commas in that command and it'll work. I fixed the docs for
> > this.
> >
> > Josh
> 
> Got no luck. osd.11.log said nothing useful. Do I have to edit
> crushmap manually? If so, how should I define host's 'id' value, I
> cannot find docs from ceph.com/docs since yesterday, where's that
> gone?
> 
> squid87-log13 ~ # ceph osd crush set 11 osd.11 3 pool=data
> datacenter=dh-1L room=room1 row=02 rack=05 host=squid87-log13

You're specifying 'pool=data', but:

> updated item id 11 name 'osd.11' weight 3 at location
> {datacenter=dh-1L,host=squid87-log13,pool=data,rack=05,room=room1,row=02}
> to crush map
> squid87-log13 ~ # ceph osd tree
> dumped osdmap tree epoch 467
> # id weight type name up/down reweight
> -1 36 pool default
> -3 36 rack unknownrack
> -2 36 host log3

the existing hierarchy has pool=default.  Change it to default above and 
you'll be okay.  You may want to restructure the existing hosts as well so 
they 'live' in the tree structure.

(This confusion is exactly why it's switched 'root=default' in the new 
releases.)

sage

> 0 3 osd.0 up 3
> 1 3 osd.1 up 3
> 2 3 osd.2 up 3
> 3 3 osd.3 up 3
> 4 3 osd.4 up 3
> 5 3 osd.5 up 3
> 6 3 osd.6 up 3
> 7 3 osd.7 up 3
> 8 3 osd.8 up 3
> 9 3 osd.9 up 3
> 10 3 osd.10 up 3
> 
> 11 0 osd.11 up 3
> 12 0 osd.12 up 3
> 13 0 osd.13 up 3
> 14 0 osd.14 up 3
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: osd not in tree

[Drunkard Zhang]

2012/11/17 Sage Weil <sage@inktank.com>:
> On Fri, 16 Nov 2012, Drunkard Zhang wrote:
>> 2012/11/16 Josh Durgin <josh.durgin@inktank.com>:
>> > On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
>> >>
>> >> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
>> >> from other hosts, But they are not in osd tree, also not usable, how
>> >> can I fix this?
>> >>
>> >> The crush command I used:
>> >> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
>> >> row=02, rack=05, host=squid87-log13
>> >
>> >
>> > Remove the commas in that command and it'll work. I fixed the docs for
>> > this.
>> >
>> > Josh
>>
>> Got no luck. osd.11.log said nothing useful. Do I have to edit
>> crushmap manually? If so, how should I define host's 'id' value, I
>> cannot find docs from ceph.com/docs since yesterday, where's that
>> gone?
>>
>> squid87-log13 ~ # ceph osd crush set 11 osd.11 3 pool=data
>> datacenter=dh-1L room=room1 row=02 rack=05 host=squid87-log13
>
> You're specifying 'pool=data', but:
>
>> updated item id 11 name 'osd.11' weight 3 at location
>> {datacenter=dh-1L,host=squid87-log13,pool=data,rack=05,room=room1,row=02}
>> to crush map
>> squid87-log13 ~ # ceph osd tree
>> dumped osdmap tree epoch 467
>> # id weight type name up/down reweight
>> -1 36 pool default
>> -3 36 rack unknownrack
>> -2 36 host log3
>
> the existing hierarchy has pool=default.  Change it to default above and
> you'll be okay.  You may want to restructure the existing hosts as well so
> they 'live' in the tree structure.
>
Still no luck. I'm using 0.51, not update yet. Set host=log3 make
osd.{11..14} usable, so I'm thinking I have to create the
host=squid87-log13 first, how could I create the host, modify
crushmap?

log3 ~ # for i in {11..14}; do ceph osd crush set $i osd.$i 3
pool=default datacenter=dh-1L room=room1 row=02 rack=rack0205
host=squid87-log13; done
updated item id 11 name 'osd.11' weight 3 at location
{datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
to crush map
updated item id 12 name 'osd.12' weight 3 at location
{datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
to crush map
updated item id 13 name 'osd.13' weight 3 at location
{datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
to crush map
updated item id 14 name 'osd.14' weight 3 at location
{datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
to crush map
log3 ~ # ceph osd tree
dumped osdmap tree epoch 559
# id weight type name up/down reweight
-1 33 pool default
-3 33 rack rack0205
-2 33 host log3
0 3 osd.0 up 3
1 3 osd.1 up 3
2 3 osd.2 up 3
3 3 osd.3 up 3
4 3 osd.4 up 3
5 3 osd.5 up 3
6 3 osd.6 up 3
7 3 osd.7 up 3
8 3 osd.8 up 3
9 3 osd.9 up 3
10 3 osd.10 up 3

11 0 osd.11 up 3
12 0 osd.12 up 3
13 0 osd.13 up 3
14 0 osd.14 up 3

> (This confusion is exactly why it's switched 'root=default' in the new
> releases.)
>
Yes, weird.
log3 ~ # ceph osd dump | grep ^pool
pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
320 pgp_num 320 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
pg_num 320 pgp_num 320 last_change 6 owner 0
pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num
320 pgp_num 320 last_change 1 owner 0

Re: osd not in tree

[Sage Weil]

On Sat, 17 Nov 2012, Drunkard Zhang wrote:
> 2012/11/17 Sage Weil <sage@inktank.com>:
> > On Fri, 16 Nov 2012, Drunkard Zhang wrote:
> >> 2012/11/16 Josh Durgin <josh.durgin@inktank.com>:
> >> > On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
> >> >>
> >> >> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
> >> >> from other hosts, But they are not in osd tree, also not usable, how
> >> >> can I fix this?
> >> >>
> >> >> The crush command I used:
> >> >> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
> >> >> row=02, rack=05, host=squid87-log13
> >> >
> >> >
> >> > Remove the commas in that command and it'll work. I fixed the docs for
> >> > this.
> >> >
> >> > Josh
> >>
> >> Got no luck. osd.11.log said nothing useful. Do I have to edit
> >> crushmap manually? If so, how should I define host's 'id' value, I
> >> cannot find docs from ceph.com/docs since yesterday, where's that
> >> gone?
> >>
> >> squid87-log13 ~ # ceph osd crush set 11 osd.11 3 pool=data
> >> datacenter=dh-1L room=room1 row=02 rack=05 host=squid87-log13
> >
> > You're specifying 'pool=data', but:
> >
> >> updated item id 11 name 'osd.11' weight 3 at location
> >> {datacenter=dh-1L,host=squid87-log13,pool=data,rack=05,room=room1,row=02}
> >> to crush map
> >> squid87-log13 ~ # ceph osd tree
> >> dumped osdmap tree epoch 467
> >> # id weight type name up/down reweight
> >> -1 36 pool default
> >> -3 36 rack unknownrack
> >> -2 36 host log3
> >
> > the existing hierarchy has pool=default.  Change it to default above and
> > you'll be okay.  You may want to restructure the existing hosts as well so
> > they 'live' in the tree structure.
> >
> Still no luck. I'm using 0.51, not update yet. Set host=log3 make
> osd.{11..14} usable, so I'm thinking I have to create the
> host=squid87-log13 first, how could I create the host, modify
> crushmap?
> 
> log3 ~ # for i in {11..14}; do ceph osd crush set $i osd.$i 3
> pool=default datacenter=dh-1L room=room1 row=02 rack=rack0205
> host=squid87-log13; done
> updated item id 11 name 'osd.11' weight 3 at location
> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> to crush map
> updated item id 12 name 'osd.12' weight 3 at location
> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> to crush map
> updated item id 13 name 'osd.13' weight 3 at location
> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> to crush map
> updated item id 14 name 'osd.14' weight 3 at location
> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> to crush map
> log3 ~ # ceph osd tree
> dumped osdmap tree epoch 559
> # id weight type name up/down reweight
> -1 33 pool default
> -3 33 rack rack0205
> -2 33 host log3
> 0 3 osd.0 up 3
> 1 3 osd.1 up 3
> 2 3 osd.2 up 3
> 3 3 osd.3 up 3
> 4 3 osd.4 up 3
> 5 3 osd.5 up 3
> 6 3 osd.6 up 3
> 7 3 osd.7 up 3
> 8 3 osd.8 up 3
> 9 3 osd.9 up 3
> 10 3 osd.10 up 3
> 
> 11 0 osd.11 up 3
> 12 0 osd.12 up 3
> 13 0 osd.13 up 3
> 14 0 osd.14 up 3

Can you do 

 ceph osd getcrushmap -o /tmp/foo
 crushtool -d /tmp/foo

and attach the output?

Thanks!
sage


> 
> > (This confusion is exactly why it's switched 'root=default' in the new
> > releases.)
> >
> Yes, weird.
> log3 ~ # ceph osd dump | grep ^pool
> pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
> 320 pgp_num 320 last_change 1 owner 0 crash_replay_interval 45
> pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
> pg_num 320 pgp_num 320 last_change 6 owner 0
> pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num
> 320 pgp_num 320 last_change 1 owner 0
> 
> 

Re: osd not in tree

[Drunkard Zhang]

2012/11/17 Sage Weil <sage@inktank.com>:
> On Sat, 17 Nov 2012, Drunkard Zhang wrote:
>> 2012/11/17 Sage Weil <sage@inktank.com>:
>> > On Fri, 16 Nov 2012, Drunkard Zhang wrote:
>> >> 2012/11/16 Josh Durgin <josh.durgin@inktank.com>:
>> >> > On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
>> >> >>
>> >> >> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
>> >> >> from other hosts, But they are not in osd tree, also not usable, how
>> >> >> can I fix this?
>> >> >>
>> >> >> The crush command I used:
>> >> >> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
>> >> >> row=02, rack=05, host=squid87-log13
>> >> >
>> >> >
>> >> > Remove the commas in that command and it'll work. I fixed the docs for
>> >> > this.
>> >> >
>> >> > Josh
>> >>
>> >> Got no luck. osd.11.log said nothing useful. Do I have to edit
>> >> crushmap manually? If so, how should I define host's 'id' value, I
>> >> cannot find docs from ceph.com/docs since yesterday, where's that
>> >> gone?
>> >>
>> >> squid87-log13 ~ # ceph osd crush set 11 osd.11 3 pool=data
>> >> datacenter=dh-1L room=room1 row=02 rack=05 host=squid87-log13
>> >
>> > You're specifying 'pool=data', but:
>> >
>> >> updated item id 11 name 'osd.11' weight 3 at location
>> >> {datacenter=dh-1L,host=squid87-log13,pool=data,rack=05,room=room1,row=02}
>> >> to crush map
>> >> squid87-log13 ~ # ceph osd tree
>> >> dumped osdmap tree epoch 467
>> >> # id weight type name up/down reweight
>> >> -1 36 pool default
>> >> -3 36 rack unknownrack
>> >> -2 36 host log3
>> >
>> > the existing hierarchy has pool=default.  Change it to default above and
>> > you'll be okay.  You may want to restructure the existing hosts as well so
>> > they 'live' in the tree structure.
>> >
>> Still no luck. I'm using 0.51, not update yet. Set host=log3 make
>> osd.{11..14} usable, so I'm thinking I have to create the
>> host=squid87-log13 first, how could I create the host, modify
>> crushmap?
>>
>> log3 ~ # for i in {11..14}; do ceph osd crush set $i osd.$i 3
>> pool=default datacenter=dh-1L room=room1 row=02 rack=rack0205
>> host=squid87-log13; done
>> updated item id 11 name 'osd.11' weight 3 at location
>> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
>> to crush map
>> updated item id 12 name 'osd.12' weight 3 at location
>> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
>> to crush map
>> updated item id 13 name 'osd.13' weight 3 at location
>> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
>> to crush map
>> updated item id 14 name 'osd.14' weight 3 at location
>> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
>> to crush map
>> log3 ~ # ceph osd tree
>> dumped osdmap tree epoch 559
>> # id weight type name up/down reweight
>> -1 33 pool default
>> -3 33 rack rack0205
>> -2 33 host log3
>> 0 3 osd.0 up 3
>> 1 3 osd.1 up 3
>> 2 3 osd.2 up 3
>> 3 3 osd.3 up 3
>> 4 3 osd.4 up 3
>> 5 3 osd.5 up 3
>> 6 3 osd.6 up 3
>> 7 3 osd.7 up 3
>> 8 3 osd.8 up 3
>> 9 3 osd.9 up 3
>> 10 3 osd.10 up 3
>>
>> 11 0 osd.11 up 3
>> 12 0 osd.12 up 3
>> 13 0 osd.13 up 3
>> 14 0 osd.14 up 3
>
> Can you do
>
>  ceph osd getcrushmap -o /tmp/foo
>  crushtool -d /tmp/foo
>
> and attach the output?
>
# begin crush map

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14

# types
type 0 osd
type 1 host
type 2 rack
type 3 row
type 4 room
type 5 datacenter
type 6 pool

# buckets
host log3 {
id -2 # do not change unnecessarily
# weight 33.000
alg straw
hash 0 # rjenkins1
item osd.0 weight 3.000
item osd.1 weight 3.000
item osd.2 weight 3.000
item osd.3 weight 3.000
item osd.4 weight 3.000
item osd.5 weight 3.000
item osd.6 weight 3.000
item osd.7 weight 3.000
item osd.8 weight 3.000
item osd.9 weight 3.000
item osd.10 weight 3.000
}
rack rack0205 {
id -3 # do not change unnecessarily
# weight 33.000
alg straw
hash 0 # rjenkins1
item log3 weight 33.000
}
pool default {
id -1 # do not change unnecessarily
# weight 33.000
alg straw
hash 0 # rjenkins1
item rack0205 weight 33.000
}
host squid87-log13 {
id -4 # do not change unnecessarily
# weight 12.000
alg straw
hash 0 # rjenkins1
item osd.12 weight 3.000
item osd.13 weight 3.000
item osd.14 weight 3.000
item osd.11 weight 3.000
}

# rules
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule metadata {
ruleset 1
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule rbd {
ruleset 2
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
# end crush map

I tried to add squid87-log13 into rack0205, but failed. Just add one
line "item squid87-log13 weight 12.000" into rack rack0205 section.

log3 ~ # crushtool -c crushmap-1117-txt -o crushmap-1117-new
item 'squid87-log13' in bucket 'rack0205' is not defined
item 'rack0205' in bucket 'default' is not defined
in rule 'data' item 'default' not defined
in rule 'metadata' item 'default' not defined
in rule 'rbd' item 'default' not defined

Re: osd not in tree

[Sage Weil]

Hi,

Okay, it looks something in the past added the host entry but for some 
reason didn't give it a parent.  Did you previously modify the crush map 
by hand, or did you only manipulate it via the 'ceph osd crush ...' 
commands?

Unfortuantely the fix is manually edit it.

ceph osd getcrushmap -o /tmp/foo
crushtool -d /tmp/foo -o /tmp/foo.txt
edit foo.txt.  remove the host bucket (squid87-log13) and all of it's 
children.
crushtool -c /tmp/foo.txt -o /tmp/foo.new
cpeh osd setcrushmap -i /tmp/foo.new

and then you can re-run those 'ceph osd crush set ...' commands and you'll 
be back in business.

I just found a bug in the 'ceph osd crush move ...' command that prevents 
us from repairing that way; that fix will be in bobtail.

sage



On Sat, 17 Nov 2012, Drunkard Zhang wrote:
> 2012/11/17 Sage Weil <sage@inktank.com>:
> > On Sat, 17 Nov 2012, Drunkard Zhang wrote:
> >> 2012/11/17 Sage Weil <sage@inktank.com>:
> >> > On Fri, 16 Nov 2012, Drunkard Zhang wrote:
> >> >> 2012/11/16 Josh Durgin <josh.durgin@inktank.com>:
> >> >> > On 11/15/2012 11:21 PM, Drunkard Zhang wrote:
> >> >> >>
> >> >> >> I installed mon x1, mds x1 and osd x11 in one host, then add some osd
> >> >> >> from other hosts, But they are not in osd tree, also not usable, how
> >> >> >> can I fix this?
> >> >> >>
> >> >> >> The crush command I used:
> >> >> >> ceph osd crush set 11 osd.11 3 pool=data datacenter=dh-1L, room=room1,
> >> >> >> row=02, rack=05, host=squid87-log13
> >> >> >
> >> >> >
> >> >> > Remove the commas in that command and it'll work. I fixed the docs for
> >> >> > this.
> >> >> >
> >> >> > Josh
> >> >>
> >> >> Got no luck. osd.11.log said nothing useful. Do I have to edit
> >> >> crushmap manually? If so, how should I define host's 'id' value, I
> >> >> cannot find docs from ceph.com/docs since yesterday, where's that
> >> >> gone?
> >> >>
> >> >> squid87-log13 ~ # ceph osd crush set 11 osd.11 3 pool=data
> >> >> datacenter=dh-1L room=room1 row=02 rack=05 host=squid87-log13
> >> >
> >> > You're specifying 'pool=data', but:
> >> >
> >> >> updated item id 11 name 'osd.11' weight 3 at location
> >> >> {datacenter=dh-1L,host=squid87-log13,pool=data,rack=05,room=room1,row=02}
> >> >> to crush map
> >> >> squid87-log13 ~ # ceph osd tree
> >> >> dumped osdmap tree epoch 467
> >> >> # id weight type name up/down reweight
> >> >> -1 36 pool default
> >> >> -3 36 rack unknownrack
> >> >> -2 36 host log3
> >> >
> >> > the existing hierarchy has pool=default.  Change it to default above and
> >> > you'll be okay.  You may want to restructure the existing hosts as well so
> >> > they 'live' in the tree structure.
> >> >
> >> Still no luck. I'm using 0.51, not update yet. Set host=log3 make
> >> osd.{11..14} usable, so I'm thinking I have to create the
> >> host=squid87-log13 first, how could I create the host, modify
> >> crushmap?
> >>
> >> log3 ~ # for i in {11..14}; do ceph osd crush set $i osd.$i 3
> >> pool=default datacenter=dh-1L room=room1 row=02 rack=rack0205
> >> host=squid87-log13; done
> >> updated item id 11 name 'osd.11' weight 3 at location
> >> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> >> to crush map
> >> updated item id 12 name 'osd.12' weight 3 at location
> >> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> >> to crush map
> >> updated item id 13 name 'osd.13' weight 3 at location
> >> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> >> to crush map
> >> updated item id 14 name 'osd.14' weight 3 at location
> >> {datacenter=dh-1L,host=squid87-log13,pool=default,rack=rack0205,room=room1,row=02}
> >> to crush map
> >> log3 ~ # ceph osd tree
> >> dumped osdmap tree epoch 559
> >> # id weight type name up/down reweight
> >> -1 33 pool default
> >> -3 33 rack rack0205
> >> -2 33 host log3
> >> 0 3 osd.0 up 3
> >> 1 3 osd.1 up 3
> >> 2 3 osd.2 up 3
> >> 3 3 osd.3 up 3
> >> 4 3 osd.4 up 3
> >> 5 3 osd.5 up 3
> >> 6 3 osd.6 up 3
> >> 7 3 osd.7 up 3
> >> 8 3 osd.8 up 3
> >> 9 3 osd.9 up 3
> >> 10 3 osd.10 up 3
> >>
> >> 11 0 osd.11 up 3
> >> 12 0 osd.12 up 3
> >> 13 0 osd.13 up 3
> >> 14 0 osd.14 up 3
> >
> > Can you do
> >
> >  ceph osd getcrushmap -o /tmp/foo
> >  crushtool -d /tmp/foo
> >
> > and attach the output?
> >
> # begin crush map
> 
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
> device 6 osd.6
> device 7 osd.7
> device 8 osd.8
> device 9 osd.9
> device 10 osd.10
> device 11 osd.11
> device 12 osd.12
> device 13 osd.13
> device 14 osd.14
> 
> # types
> type 0 osd
> type 1 host
> type 2 rack
> type 3 row
> type 4 room
> type 5 datacenter
> type 6 pool
> 
> # buckets
> host log3 {
> id -2 # do not change unnecessarily
> # weight 33.000
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 3.000
> item osd.1 weight 3.000
> item osd.2 weight 3.000
> item osd.3 weight 3.000
> item osd.4 weight 3.000
> item osd.5 weight 3.000
> item osd.6 weight 3.000
> item osd.7 weight 3.000
> item osd.8 weight 3.000
> item osd.9 weight 3.000
> item osd.10 weight 3.000
> }
> rack rack0205 {
> id -3 # do not change unnecessarily
> # weight 33.000
> alg straw
> hash 0 # rjenkins1
> item log3 weight 33.000
> }
> pool default {
> id -1 # do not change unnecessarily
> # weight 33.000
> alg straw
> hash 0 # rjenkins1
> item rack0205 weight 33.000
> }
> host squid87-log13 {
> id -4 # do not change unnecessarily
> # weight 12.000
> alg straw
> hash 0 # rjenkins1
> item osd.12 weight 3.000
> item osd.13 weight 3.000
> item osd.14 weight 3.000
> item osd.11 weight 3.000
> }
> 
> # rules
> rule data {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
> rule metadata {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
> rule rbd {
> ruleset 2
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
> # end crush map
> 
> I tried to add squid87-log13 into rack0205, but failed. Just add one
> line "item squid87-log13 weight 12.000" into rack rack0205 section.
> 
> log3 ~ # crushtool -c crushmap-1117-txt -o crushmap-1117-new
> item 'squid87-log13' in bucket 'rack0205' is not defined
> item 'rack0205' in bucket 'default' is not defined
> in rule 'data' item 'default' not defined
> in rule 'metadata' item 'default' not defined
> in rule 'rbd' item 'default' not defined
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: osd not in tree

[Drunkard Zhang]

2012/11/17 Sage Weil <sage@inktank.com>:
> Hi,
>
> Okay, it looks something in the past added the host entry but for some
> reason didn't give it a parent.  Did you previously modify the crush map
> by hand, or did you only manipulate it via the 'ceph osd crush ...'
> commands?
>
> Unfortuantely the fix is manually edit it.
>
> ceph osd getcrushmap -o /tmp/foo
> crushtool -d /tmp/foo -o /tmp/foo.txt
> edit foo.txt.  remove the host bucket (squid87-log13) and all of it's
> children.
> crushtool -c /tmp/foo.txt -o /tmp/foo.new
> cpeh osd setcrushmap -i /tmp/foo.new
>
> and then you can re-run those 'ceph osd crush set ...' commands and you'll
> be back in business.
>
Great, it works. now I remember I was trying to add host by hand while
'ceph osd crush set ...' failed (maybe command is wrong).

Another problem: is there any chance to change 'rack'? Or how to
create a new rack? If I have to create a new rack first, I made
mistake, or maybe buggy again.

Before change rack:
log3 ~ # ceph osd tree
dumped osdmap tree epoch 611
# id weight type name up/down reweight
-1 45 pool default
-3 45 rack rack0205
-2 33 host log3
0 3 osd.0 up 3
1 3 osd.1 up 3
2 3 osd.2 up 3
3 3 osd.3 up 3
4 3 osd.4 up 3
5 3 osd.5 up 3
6 3 osd.6 up 3
7 3 osd.7 up 3
8 3 osd.8 up 3
9 3 osd.9 up 3
10 3 osd.10 up 3
-4 12 host squid87-log13
11 3 osd.11 up 3
12 3 osd.12 up 3
13 3 osd.13 up 3
14 3 osd.14 up 3

Looks reasonable, but log3 miss set rack, I wanna change rack=rack0205
to rack=rack0206),  reset use this command, but not works:
for i in {0..10}; do ceph osd crush set $i osd.$i 3 pool=data
datacenter=dh-1L room=room1 row=02 rack=rack0206 host=log3; done

Re: osd not in tree

[Sage Weil]

On Sat, 17 Nov 2012, Drunkard Zhang wrote:
> 2012/11/17 Sage Weil <sage@inktank.com>:
> > Hi,
> >
> > Okay, it looks something in the past added the host entry but for some
> > reason didn't give it a parent.  Did you previously modify the crush map
> > by hand, or did you only manipulate it via the 'ceph osd crush ...'
> > commands?
> >
> > Unfortuantely the fix is manually edit it.
> >
> > ceph osd getcrushmap -o /tmp/foo
> > crushtool -d /tmp/foo -o /tmp/foo.txt
> > edit foo.txt.  remove the host bucket (squid87-log13) and all of it's
> > children.
> > crushtool -c /tmp/foo.txt -o /tmp/foo.new
> > cpeh osd setcrushmap -i /tmp/foo.new
> >
> > and then you can re-run those 'ceph osd crush set ...' commands and you'll
> > be back in business.
> >
> Great, it works. now I remember I was trying to add host by hand while
> 'ceph osd crush set ...' failed (maybe command is wrong).
> 
> Another problem: is there any chance to change 'rack'? Or how to
> create a new rack? If I have to create a new rack first, I made
> mistake, or maybe buggy again.
> 
> Before change rack:
> log3 ~ # ceph osd tree
> dumped osdmap tree epoch 611
> # id weight type name up/down reweight
> -1 45 pool default
> -3 45 rack rack0205
> -2 33 host log3
> 0 3 osd.0 up 3
> 1 3 osd.1 up 3
> 2 3 osd.2 up 3
> 3 3 osd.3 up 3
> 4 3 osd.4 up 3
> 5 3 osd.5 up 3
> 6 3 osd.6 up 3
> 7 3 osd.7 up 3
> 8 3 osd.8 up 3
> 9 3 osd.9 up 3
> 10 3 osd.10 up 3
> -4 12 host squid87-log13
> 11 3 osd.11 up 3
> 12 3 osd.12 up 3
> 13 3 osd.13 up 3
> 14 3 osd.14 up 3
> 
> Looks reasonable, but log3 miss set rack, I wanna change rack=rack0205
> to rack=rack0206),  reset use this command, but not works:
> for i in {0..10}; do ceph osd crush set $i osd.$i 3 pool=data
> datacenter=dh-1L room=room1 row=02 rack=rack0206 host=log3; done

'ceph osd crush set ...' will only move the device itself; it won't move 
any of its parents.  If you want to move a non-leaf item in the tree, use 
'ceph osd crush move <name> <location ...>'.  Something like

 ceph osd crush move squid87-log13 rack=0206 pool=default
 ceph osd crush move log3 rack=0206 pool=default

sage

Re: osd not in tree

[Drunkard Zhang]

2012/11/17 Sage Weil <sage@inktank.com>:
> On Sat, 17 Nov 2012, Drunkard Zhang wrote:
>> 2012/11/17 Sage Weil <sage@inktank.com>:
>> > Hi,
>> >
>> > Okay, it looks something in the past added the host entry but for some
>> > reason didn't give it a parent.  Did you previously modify the crush map
>> > by hand, or did you only manipulate it via the 'ceph osd crush ...'
>> > commands?
>> >
>> > Unfortuantely the fix is manually edit it.
>> >
>> > ceph osd getcrushmap -o /tmp/foo
>> > crushtool -d /tmp/foo -o /tmp/foo.txt
>> > edit foo.txt.  remove the host bucket (squid87-log13) and all of it's
>> > children.
>> > crushtool -c /tmp/foo.txt -o /tmp/foo.new
>> > cpeh osd setcrushmap -i /tmp/foo.new
>> >
>> > and then you can re-run those 'ceph osd crush set ...' commands and you'll
>> > be back in business.
>> >
>> Great, it works. now I remember I was trying to add host by hand while
>> 'ceph osd crush set ...' failed (maybe command is wrong).
>>
>> Another problem: is there any chance to change 'rack'? Or how to
>> create a new rack? If I have to create a new rack first, I made
>> mistake, or maybe buggy again.
>>
>> Before change rack:
>> log3 ~ # ceph osd tree
>> dumped osdmap tree epoch 611
>> # id weight type name up/down reweight
>> -1 45 pool default
>> -3 45 rack rack0205
>> -2 33 host log3
>> 0 3 osd.0 up 3
>> 1 3 osd.1 up 3
>> 2 3 osd.2 up 3
>> 3 3 osd.3 up 3
>> 4 3 osd.4 up 3
>> 5 3 osd.5 up 3
>> 6 3 osd.6 up 3
>> 7 3 osd.7 up 3
>> 8 3 osd.8 up 3
>> 9 3 osd.9 up 3
>> 10 3 osd.10 up 3
>> -4 12 host squid87-log13
>> 11 3 osd.11 up 3
>> 12 3 osd.12 up 3
>> 13 3 osd.13 up 3
>> 14 3 osd.14 up 3
>>
>> Looks reasonable, but log3 miss set rack, I wanna change rack=rack0205
>> to rack=rack0206),  reset use this command, but not works:
>> for i in {0..10}; do ceph osd crush set $i osd.$i 3 pool=data
>> datacenter=dh-1L room=room1 row=02 rack=rack0206 host=log3; done
>
> 'ceph osd crush set ...' will only move the device itself; it won't move
> any of its parents.  If you want to move a non-leaf item in the tree, use
> 'ceph osd crush move <name> <location ...>'.  Something like
>
>  ceph osd crush move squid87-log13 rack=0206 pool=default
>  ceph osd crush move log3 rack=0206 pool=default
>
> sage


Thank you a lot. I'll keep digging :-)