problem with removing osd

problem with removing osd

[Łukasz Chrustek]

Hi,

I was trying to delete 3 osds from cluster, deletion procces took very
long  time and I interrupted it. mon process then crushed, and in ceph
osd tree (after restart ceph-mon) I saw:

 ~]# ceph osd tree
ID         WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
        -7  16.89590 root ssd-disks
       -11         0     host ssd1
-231707408         0
     22100         0         osd.22100        DNE        0
        71         0         osd.71           DNE        0


when I tried to delete osd.22100:

[root@cc1 ~]# ceph osd crush remove osd.22100
device 'osd.22100' does not appear in the crush map

then I tried to delete osd.71 and mon proccess crushed:

[root@cc1 ~]# ceph osd crush remove osd.71
2016-12-28 17:52:34.459668 7f426a862700  0 monclient: hunting for new mon

after restart of ceph-mon in ceph osd tree it shows:

# ceph osd tree
ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
       -7  16.89590 root ssd-disks
      -11         0     host ssd1
598798032         0         osd.598798032     DNE        0
    21940         0         osd.21940         DNE        0
       71         0         osd.71            DNE        0

My question is how to delete this osds without direct editing crushmap
? It is production system, I can't affort any service interruption :(,
when I try to ceph osd crush remove then ceph-mon crushes....

I  dumped  crushmap,  but it took 19G (!!) after decompiling (compiled
file  is  very small). So, I cleaned this file with perl (it take very
long  time), and I have now small txt crushmap, which I edited. But is
there  any  chance  that ceph will still remember somewhere about this
huge  numbers  for osds ? Is it safe to apply this cleaned crushmap to
cluster ?

-- 
Regards
Luk

Re: problem with removing osd

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2244 bytes --]

On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
> Hi,
> 
> I was trying to delete 3 osds from cluster, deletion procces took very
> long  time and I interrupted it. mon process then crushed, and in ceph
> osd tree (after restart ceph-mon) I saw:
> 
>  ~]# ceph osd tree
> ID         WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
>         -7  16.89590 root ssd-disks
>        -11         0     host ssd1
> -231707408         0
>      22100         0         osd.22100        DNE        0
>         71         0         osd.71           DNE        0
> 
> 
> when I tried to delete osd.22100:
> 
> [root@cc1 ~]# ceph osd crush remove osd.22100
> device 'osd.22100' does not appear in the crush map
> 
> then I tried to delete osd.71 and mon proccess crushed:
> 
> [root@cc1 ~]# ceph osd crush remove osd.71
> 2016-12-28 17:52:34.459668 7f426a862700  0 monclient: hunting for new mon
> 
> after restart of ceph-mon in ceph osd tree it shows:
> 
> # ceph osd tree
> ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
>        -7  16.89590 root ssd-disks
>       -11         0     host ssd1
> 598798032         0         osd.598798032     DNE        0

Yikes!

>     21940         0         osd.21940         DNE        0
>        71         0         osd.71            DNE        0
> 
> My question is how to delete this osds without direct editing crushmap
> ? It is production system, I can't affort any service interruption :(,
> when I try to ceph osd crush remove then ceph-mon crushes....
> 
> I  dumped  crushmap,  but it took 19G (!!) after decompiling (compiled
> file  is  very small). So, I cleaned this file with perl (it take very
> long  time), and I have now small txt crushmap, which I edited. But is
> there  any  chance  that ceph will still remember somewhere about this
> huge  numbers  for osds ? Is it safe to apply this cleaned crushmap to
> cluster ?

It sounds like the problem is the OSDMap, not CRUSH per se.  Can you 
attach the output from 'ceph osd dump -f json-pretty'?

Do you know how osd.598798032 got created?  Or osd.21940 for that matter.  
OSD ids should be small since they are stored internally by OSDMap as a 
vector.  This is probably why your mon is crashing.

sage

Re: problem with removing osd

[Łukasz Chrustek]

Hi,


>> 
>> # ceph osd tree
>> ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>        -7  16.89590 root ssd-disks
>>       -11         0     host ssd1
>> 598798032         0         osd.598798032     DNE        0

> Yikes!

Yes... indeed, I don't like this number too...

>>     21940         0         osd.21940         DNE        0
>>        71         0         osd.71            DNE        0
>> 
>> My question is how to delete this osds without direct editing crushmap
>> ? It is production system, I can't affort any service interruption :(,
>> when I try to ceph osd crush remove then ceph-mon crushes....
>> 
>> I  dumped  crushmap,  but it took 19G (!!) after decompiling (compiled
>> file  is  very small). So, I cleaned this file with perl (it take very
>> long  time), and I have now small txt crushmap, which I edited. But is
>> there  any  chance  that ceph will still remember somewhere about this
>> huge  numbers  for osds ? Is it safe to apply this cleaned crushmap to
>> cluster ?

> It sounds like the problem is the OSDMap, not CRUSH per se.  Can you 
> attach the output from 'ceph osd dump -f json-pretty'?

It's quite big so I put it on pastebin:

http://pastebin.com/Unkk2Pa7

> Do you know how osd.598798032 got created?  Or osd.21940 for that matter.
> OSD ids should be small since they are stored internally by OSDMap as a
> vector.  This is probably why your mon is crashing.

[root@cc1 /etc/ceph]# ceph osd tree
ID  WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
 -7  16.89590 root ssd-intel-s3700
-11         0     host ssd-stor1
 69         0         osd.69          down        0          1.00000
 70         0         osd.70          down        0          1.00000
 71         0         osd.71          down        0          1.00000


This the moment, when it happend:
]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done
removed item id 69 name 'osd.69' from crush map


removed item id 70 name 'osd.70' from crush map

here i press ctrl+c

2016-12-28 17:38:10.055239 7f4576d7a700  0 monclient: hunting for new mon
2016-12-28 17:38:10.055582 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault
2016-12-28 17:38:30.550622 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault
2016-12-28 17:38:54.551031 7f4574474700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault

after restart of ceph-mon:

]# ceph osd tree
ID         WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
        -7  16.89590 root ssd-intel-s3700
       -11         0     host ssd-stor1
-231707408         0
     22100         0         osd.22100        DNE        0
        71         0         osd.71           DNE        0

and later:

[root@cc1 ~]# ceph osd crush remove osd.22100
device 'osd.22100' does not appear in the crush map
[root@cc1 ~]# ceph osd crush remove osd.71
2016-12-28 17:52:34.459668 7f426a862700  0 monclient: hunting for new mon
2016-12-28 17:52:55.238418 7f426a862700  0 monclient: hunting for new mon
2016-12-28 17:52:55.238680 7f4262ebc700  0 -- 192.168.128.1:0/692048545 >> 192.168.128.2:6789/0 pipe(0x7f4254028300 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4254026800).fault

and after another restart of ceph-mon:

]# ceph osd tree
ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
       -7  16.89590 root ssd-intel-s3700
      -11         0     host ssd-stor1
598798032         0         osd.598798032     DNE        0
    21940         0         osd.21940         DNE        0
       71         0         osd.71            DNE        0




-- 
Regards
Luk

Re: problem with removing osd

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4393 bytes --]

On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
> Hi,
> 
> 
> >> 
> >> # ceph osd tree
> >> ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >>        -7  16.89590 root ssd-disks
> >>       -11         0     host ssd1
> >> 598798032         0         osd.598798032     DNE        0
> 
> > Yikes!
> 
> Yes... indeed, I don't like this number too...
> 
> >>     21940         0         osd.21940         DNE        0
> >>        71         0         osd.71            DNE        0
> >> 
> >> My question is how to delete this osds without direct editing crushmap
> >> ? It is production system, I can't affort any service interruption :(,
> >> when I try to ceph osd crush remove then ceph-mon crushes....
> >> 
> >> I  dumped  crushmap,  but it took 19G (!!) after decompiling (compiled
> >> file  is  very small). So, I cleaned this file with perl (it take very
> >> long  time), and I have now small txt crushmap, which I edited. But is
> >> there  any  chance  that ceph will still remember somewhere about this
> >> huge  numbers  for osds ? Is it safe to apply this cleaned crushmap to
> >> cluster ?
> 
> > It sounds like the problem is the OSDMap, not CRUSH per se.  Can you 
> > attach the output from 'ceph osd dump -f json-pretty'?
> 
> It's quite big so I put it on pastebin:
> 
> http://pastebin.com/Unkk2Pa7
> 
> > Do you know how osd.598798032 got created?  Or osd.21940 for that matter.
> > OSD ids should be small since they are stored internally by OSDMap as a
> > vector.  This is probably why your mon is crashing.
> 
> [root@cc1 /etc/ceph]# ceph osd tree
> ID  WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
>  -7  16.89590 root ssd-intel-s3700
> -11         0     host ssd-stor1
>  69         0         osd.69          down        0          1.00000
>  70         0         osd.70          down        0          1.00000
>  71         0         osd.71          down        0          1.00000
> 
> 
> This the moment, when it happend:
> ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done
> removed item id 69 name 'osd.69' from crush map
> 
> 
> removed item id 70 name 'osd.70' from crush map
> 
> here i press ctrl+c
> 
> 2016-12-28 17:38:10.055239 7f4576d7a700  0 monclient: hunting for new mon
> 2016-12-28 17:38:10.055582 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault
> 2016-12-28 17:38:30.550622 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault
> 2016-12-28 17:38:54.551031 7f4574474700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault

What version is this?

Can you attach the crush map too?  (ceph osd crush dump -f json-pretty)

Thanks!
sage

> after restart of ceph-mon:
> 
> ]# ceph osd tree
> ID         WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
>         -7  16.89590 root ssd-intel-s3700
>        -11         0     host ssd-stor1
> -231707408         0
>      22100         0         osd.22100        DNE        0
>         71         0         osd.71           DNE        0
> 
> and later:
> 
> [root@cc1 ~]# ceph osd crush remove osd.22100
> device 'osd.22100' does not appear in the crush map
> [root@cc1 ~]# ceph osd crush remove osd.71
> 2016-12-28 17:52:34.459668 7f426a862700  0 monclient: hunting for new mon
> 2016-12-28 17:52:55.238418 7f426a862700  0 monclient: hunting for new mon
> 2016-12-28 17:52:55.238680 7f4262ebc700  0 -- 192.168.128.1:0/692048545 >> 192.168.128.2:6789/0 pipe(0x7f4254028300 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4254026800).fault
> 
> and after another restart of ceph-mon:
> 
> ]# ceph osd tree
> ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
>        -7  16.89590 root ssd-intel-s3700
>       -11         0     host ssd-stor1
> 598798032         0         osd.598798032     DNE        0
>     21940         0         osd.21940         DNE        0
>        71         0         osd.71            DNE        0
> 
> 
> 
> 
> -- 
> Regards
> Luk
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: problem with removing osd

[Łukasz Chrustek]

Cześć,

> On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
>> Hi,
>> 
>> 
>> >> 
>> >> # ceph osd tree
>> >> ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
>> >>        -7  16.89590 root ssd-disks
>> >>       -11         0     host ssd1
>> >> 598798032         0         osd.598798032     DNE        0
>> 
>> > Yikes!
>> 
>> Yes... indeed, I don't like this number too...
>> 
>> >>     21940         0         osd.21940         DNE        0
>> >>        71         0         osd.71            DNE        0
>> >> 
>> >> My question is how to delete this osds without direct editing crushmap
>> >> ? It is production system, I can't affort any service interruption :(,
>> >> when I try to ceph osd crush remove then ceph-mon crushes....
>> >> 
>> >> I  dumped  crushmap,  but it took 19G (!!) after decompiling (compiled
>> >> file  is  very small). So, I cleaned this file with perl (it take very
>> >> long  time), and I have now small txt crushmap, which I edited. But is
>> >> there  any  chance  that ceph will still remember somewhere about this
>> >> huge  numbers  for osds ? Is it safe to apply this cleaned crushmap to
>> >> cluster ?
>> 
>> > It sounds like the problem is the OSDMap, not CRUSH per se.  Can you 
>> > attach the output from 'ceph osd dump -f json-pretty'?
>> 
>> It's quite big so I put it on pastebin:
>> 
>> http://pastebin.com/Unkk2Pa7
>> 
>> > Do you know how osd.598798032 got created?  Or osd.21940 for that matter.
>> > OSD ids should be small since they are stored internally by OSDMap as a
>> > vector.  This is probably why your mon is crashing.
>> 
>> [root@cc1 /etc/ceph]# ceph osd tree
>> ID  WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>  -7  16.89590 root ssd-intel-s3700
>> -11         0     host ssd-stor1
>>  69         0         osd.69          down        0          1.00000
>>  70         0         osd.70          down        0          1.00000
>>  71         0         osd.71          down        0          1.00000
>> 
>> 
>> This the moment, when it happend:
>> ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done
>> removed item id 69 name 'osd.69' from crush map
>> 
>> 
>> removed item id 70 name 'osd.70' from crush map
>> 
>> here i press ctrl+c
>> 
>> 2016-12-28 17:38:10.055239 7f4576d7a700  0 monclient: hunting for new mon
>> 2016-12-28 17:38:10.055582 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault
>> 2016-12-28 17:38:30.550622 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault
>> 2016-12-28 17:38:54.551031 7f4574474700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault

> What version is this?

infernalis

> Can you attach the crush map too?  (ceph osd crush dump -f json-pretty)

I can't - ceph-mons are crushing on diffrent ceph-mon hosts:

]# ceph osd crush dump -f json-pretty

2016-12-29 21:42:47.172097 7f0c36061700  0 monclient: hunting for new mon
^CTraceback (most recent call last):
  File "/usr/bin/ceph", line 948, in <module>
    retval = main()
  File "/usr/bin/ceph", line 881, in main
    sigdict, inbuf, verbose)
  File "/usr/bin/ceph", line 479, in new_style_command
    inbuf=inbuf)
  File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1291, in json_command
    raise RuntimeError('"{0}": exception {1}'.format(argdict, e))
RuntimeError: "{'prefix': u'osd crush dump', 'format': 'json-pretty'}": exception "['{"prefix": "osd crush dump", "format": "json-pretty"}']": exception 'int' object is not iterable

-- 
Pozdrowienia,
 Łukasz Chrustek

Re: problem with removing osd

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3336 bytes --]

On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
> Cześć,
> 
> > On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
> >> Hi,
> >> 
> >> 
> >> >> 
> >> >> # ceph osd tree
> >> >> ID        WEIGHT    TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >> >>        -7  16.89590 root ssd-disks
> >> >>       -11         0     host ssd1
> >> >> 598798032         0         osd.598798032     DNE        0
> >> 
> >> > Yikes!
> >> 
> >> Yes... indeed, I don't like this number too...
> >> 
> >> >>     21940         0         osd.21940         DNE        0
> >> >>        71         0         osd.71            DNE        0
> >> >> 
> >> >> My question is how to delete this osds without direct editing crushmap
> >> >> ? It is production system, I can't affort any service interruption :(,
> >> >> when I try to ceph osd crush remove then ceph-mon crushes....
> >> >> 
> >> >> I  dumped  crushmap,  but it took 19G (!!) after decompiling (compiled
> >> >> file  is  very small). So, I cleaned this file with perl (it take very
> >> >> long  time), and I have now small txt crushmap, which I edited. But is
> >> >> there  any  chance  that ceph will still remember somewhere about this
> >> >> huge  numbers  for osds ? Is it safe to apply this cleaned crushmap to
> >> >> cluster ?
> >> 
> >> > It sounds like the problem is the OSDMap, not CRUSH per se.  Can you 
> >> > attach the output from 'ceph osd dump -f json-pretty'?
> >> 
> >> It's quite big so I put it on pastebin:
> >> 
> >> http://pastebin.com/Unkk2Pa7
> >> 
> >> > Do you know how osd.598798032 got created?  Or osd.21940 for that matter.
> >> > OSD ids should be small since they are stored internally by OSDMap as a
> >> > vector.  This is probably why your mon is crashing.
> >> 
> >> [root@cc1 /etc/ceph]# ceph osd tree
> >> ID  WEIGHT    TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >>  -7  16.89590 root ssd-intel-s3700
> >> -11         0     host ssd-stor1
> >>  69         0         osd.69          down        0          1.00000
> >>  70         0         osd.70          down        0          1.00000
> >>  71         0         osd.71          down        0          1.00000
> >> 
> >> 
> >> This the moment, when it happend:
> >> ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done
> >> removed item id 69 name 'osd.69' from crush map
> >> 
> >> 
> >> removed item id 70 name 'osd.70' from crush map
> >> 
> >> here i press ctrl+c
> >> 
> >> 2016-12-28 17:38:10.055239 7f4576d7a700  0 monclient: hunting for new mon
> >> 2016-12-28 17:38:10.055582 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault
> >> 2016-12-28 17:38:30.550622 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault
> >> 2016-12-28 17:38:54.551031 7f4574474700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault
> 
> > What version is this?
> 
> infernalis
> 
> > Can you attach the crush map too?  (ceph osd crush dump -f json-pretty)
> 
> I can't - ceph-mons are crushing on diffrent ceph-mon hosts:
> 
> ]# ceph osd crush dump -f json-pretty

Hmm, in that case, 'ceph osd getcrushmap -o cm' and post that somewhere?

sage

Re: problem with removing osd

[Łukasz Chrustek]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Cześć,


>> >> 
>> >> here i press ctrl+c
>> >> 
>> >> 2016-12-28 17:38:10.055239 7f4576d7a700  0 monclient: hunting for new mon
>> >> 2016-12-28 17:38:10.055582 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault
>> >> 2016-12-28 17:38:30.550622 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault
>> >> 2016-12-28 17:38:54.551031 7f4574474700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault
>> 
>> > What version is this?
>> 
>> infernalis
>> 
>> > Can you attach the crush map too?  (ceph osd crush dump -f json-pretty)
>> 
>> I can't - ceph-mons are crushing on diffrent ceph-mon hosts:
>> 
>> ]# ceph osd crush dump -f json-pretty

> Hmm, in that case, 'ceph osd getcrushmap -o cm' and post that somewhere?

In attachment.


> sage



-- 
Regards
Luk

[-- Attachment #2: cm --]
[-- Type: application/octet-stream, Size: 3559 bytes --]

Re: problem with removing osd

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1660 bytes --]

On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
> Cześć,
> 
> 
> >> >> 
> >> >> here i press ctrl+c
> >> >> 
> >> >> 2016-12-28 17:38:10.055239 7f4576d7a700  0 monclient: hunting for new mon
> >> >> 2016-12-28 17:38:10.055582 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault
> >> >> 2016-12-28 17:38:30.550622 7f4574233700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault
> >> >> 2016-12-28 17:38:54.551031 7f4574474700  0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault
> >> 
> >> > What version is this?
> >> 
> >> infernalis
> >> 
> >> > Can you attach the crush map too?  (ceph osd crush dump -f json-pretty)
> >> 
> >> I can't - ceph-mons are crushing on diffrent ceph-mon hosts:
> >> 
> >> ]# ceph osd crush dump -f json-pretty
> 
> > Hmm, in that case, 'ceph osd getcrushmap -o cm' and post that somewhere?
> 
> In attachment.

The encoded tree bucket -11 had bad values.  I don't really trust the tree 
bucket code in crush... it's not well tested (and is a poor balance 
computation and efficiency anyway).  We should probably try to remove tree 
entirely.

I've attached a fixed map that you can inject with

 ceph osd setcrushmap -i <filename>

Bucket -11 is now empty; not sure what was supposed to be in it.

I suggest switching all of your tree buckets over to straw2 as soon as 
possible.  Note that this will result in some rebalancing.  You could do 
it one bucket a time if that's concerning.

sage

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 3627 bytes --]

Re: problem with removing osd

[Łukasz Chrustek]

Hi,

Thank  You  very  much for analize and the file ! I had similar :) but
wasn't sure if it wan't distroy something in cluster.

> The encoded tree bucket -11 had bad values.  I don't really trust the tree
> bucket code in crush... it's not well tested (and is a poor balance 
> computation and efficiency anyway).  We should probably try to remove tree
> entirely.

> I've attached a fixed map that you can inject with

>  ceph osd setcrushmap -i <filename>

Now it works, and also ceph osd crush dump -f json-pretty runs OK.

> Bucket -11 is now empty; not sure what was supposed to be in it.

this server will be reinstalled, there where three osds.

> I suggest switching all of your tree buckets over to straw2 as soon as
> possible.  Note that this will result in some rebalancing.  You could do
> it one bucket a time if that's concerning.

OK,  changing  alg  to straw2 will rebalance ale PGs on all nodes ?
-- 
Regards,
Luk

Re: problem with removing osd

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1366 bytes --]

On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
> Hi,
> 
> Thank  You  very  much for analize and the file ! I had similar :) but
> wasn't sure if it wan't distroy something in cluster.
> 
> > The encoded tree bucket -11 had bad values.  I don't really trust the tree
> > bucket code in crush... it's not well tested (and is a poor balance 
> > computation and efficiency anyway).  We should probably try to remove tree
> > entirely.
> 
> > I've attached a fixed map that you can inject with
> 
> >  ceph osd setcrushmap -i <filename>
> 
> Now it works, and also ceph osd crush dump -f json-pretty runs OK.

Great news!
 
> > Bucket -11 is now empty; not sure what was supposed to be in it.
> 
> this server will be reinstalled, there where three osds.
> 
> > I suggest switching all of your tree buckets over to straw2 as soon as
> > possible.  Note that this will result in some rebalancing.  You could do
> > it one bucket a time if that's concerning.
> 
> OK,  changing  alg  to straw2 will rebalance ale PGs on all nodes ?

For any bucket you change from tree -> straw2, you'll see PGs shuffle 
between the children of that bucket.  So for hosts, you'll see data move 
between the disks.  And fixing ssd-intel-s3700 will shuffle data between 
hosts.

I'd also switch the straw buckets to straw2, although that will move a 
comparatively small amount of data.

sage

Re: problem with removing osd

[Brad Hubbard]

Have you thought through your use of '"min_size": 1' on many of these pools?


See recent discussions on this mailing list on the subject for background.

Just FYI.


On Fri, Dec 30, 2016 at 8:18 AM, Sage Weil <sage@newdream.net> wrote:
> On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
>> Hi,
>>
>> Thank  You  very  much for analize and the file ! I had similar :) but
>> wasn't sure if it wan't distroy something in cluster.
>>
>> > The encoded tree bucket -11 had bad values.  I don't really trust the tree
>> > bucket code in crush... it's not well tested (and is a poor balance
>> > computation and efficiency anyway).  We should probably try to remove tree
>> > entirely.
>>
>> > I've attached a fixed map that you can inject with
>>
>> >  ceph osd setcrushmap -i <filename>
>>
>> Now it works, and also ceph osd crush dump -f json-pretty runs OK.
>
> Great news!
>
>> > Bucket -11 is now empty; not sure what was supposed to be in it.
>>
>> this server will be reinstalled, there where three osds.
>>
>> > I suggest switching all of your tree buckets over to straw2 as soon as
>> > possible.  Note that this will result in some rebalancing.  You could do
>> > it one bucket a time if that's concerning.
>>
>> OK,  changing  alg  to straw2 will rebalance ale PGs on all nodes ?
>
> For any bucket you change from tree -> straw2, you'll see PGs shuffle
> between the children of that bucket.  So for hosts, you'll see data move
> between the disks.  And fixing ssd-intel-s3700 will shuffle data between
> hosts.
>
> I'd also switch the straw buckets to straw2, although that will move a
> comparatively small amount of data.
>
> sage



-- 
Cheers,
Brad

Re: problem with removing osd

[Łukasz Chrustek]

Hi,

Thank You for suggestion - I will read about it.

Regards
Luk

> Have you thought through your use of '"min_size": 1' on many of these pools?


> See recent discussions on this mailing list on the subject for background.

> Just FYI.


> On Fri, Dec 30, 2016 at 8:18 AM, Sage Weil <sage@newdream.net> wrote:
>> On Thu, 29 Dec 2016, Łukasz Chrustek wrote:
>>> Hi,
>>>
>>> Thank  You  very  much for analize and the file ! I had similar :) but
>>> wasn't sure if it wan't distroy something in cluster.
>>>
>>> > The encoded tree bucket -11 had bad values.  I don't really trust the tree
>>> > bucket code in crush... it's not well tested (and is a poor balance
>>> > computation and efficiency anyway).  We should probably try to remove tree
>>> > entirely.
>>>
>>> > I've attached a fixed map that you can inject with
>>>
>>> >  ceph osd setcrushmap -i <filename>
>>>
>>> Now it works, and also ceph osd crush dump -f json-pretty runs OK.
>>
>> Great news!
>>
>>> > Bucket -11 is now empty; not sure what was supposed to be in it.
>>>
>>> this server will be reinstalled, there where three osds.
>>>
>>> > I suggest switching all of your tree buckets over to straw2 as soon as
>>> > possible.  Note that this will result in some rebalancing.  You could do
>>> > it one bucket a time if that's concerning.
>>>
>>> OK,  changing  alg  to straw2 will rebalance ale PGs on all nodes ?
>>
>> For any bucket you change from tree -> straw2, you'll see PGs shuffle
>> between the children of that bucket.  So for hosts, you'll see data move
>> between the disks.  And fixing ssd-intel-s3700 will shuffle data between
>> hosts.
>>
>> I'd also switch the straw buckets to straw2, although that will move a
>> comparatively small amount of data.
>>
>> sage






-- 
Pozdrowienia,
 Łukasz Chrustek