* problem with removing osd
@ 2016-12-29 15:07 Łukasz Chrustek
2016-12-29 19:10 ` Sage Weil
0 siblings, 1 reply; 12+ messages in thread
From: Łukasz Chrustek @ 2016-12-29 15:07 UTC (permalink / raw)
To: ceph-devel
Hi,
I was trying to delete 3 osds from cluster, deletion procces took very
long time and I interrupted it. mon process then crushed, and in ceph
osd tree (after restart ceph-mon) I saw:
~]# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-7 16.89590 root ssd-disks
-11 0 host ssd1
-231707408 0
22100 0 osd.22100 DNE 0
71 0 osd.71 DNE 0
when I tried to delete osd.22100:
[root@cc1 ~]# ceph osd crush remove osd.22100
device 'osd.22100' does not appear in the crush map
then I tried to delete osd.71 and mon proccess crushed:
[root@cc1 ~]# ceph osd crush remove osd.71
2016-12-28 17:52:34.459668 7f426a862700 0 monclient: hunting for new mon
after restart of ceph-mon in ceph osd tree it shows:
# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-7 16.89590 root ssd-disks
-11 0 host ssd1
598798032 0 osd.598798032 DNE 0
21940 0 osd.21940 DNE 0
71 0 osd.71 DNE 0
My question is how to delete this osds without direct editing crushmap
? It is production system, I can't affort any service interruption :(,
when I try to ceph osd crush remove then ceph-mon crushes....
I dumped crushmap, but it took 19G (!!) after decompiling (compiled
file is very small). So, I cleaned this file with perl (it take very
long time), and I have now small txt crushmap, which I edited. But is
there any chance that ceph will still remember somewhere about this
huge numbers for osds ? Is it safe to apply this cleaned crushmap to
cluster ?
--
Regards
Luk
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: problem with removing osd 2016-12-29 15:07 problem with removing osd Łukasz Chrustek @ 2016-12-29 19:10 ` Sage Weil 2016-12-29 20:20 ` Łukasz Chrustek 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2016-12-29 19:10 UTC (permalink / raw) To: Łukasz Chrustek; +Cc: ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 2244 bytes --] On Thu, 29 Dec 2016, Łukasz Chrustek wrote: > Hi, > > I was trying to delete 3 osds from cluster, deletion procces took very > long time and I interrupted it. mon process then crushed, and in ceph > osd tree (after restart ceph-mon) I saw: > > ~]# ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -7 16.89590 root ssd-disks > -11 0 host ssd1 > -231707408 0 > 22100 0 osd.22100 DNE 0 > 71 0 osd.71 DNE 0 > > > when I tried to delete osd.22100: > > [root@cc1 ~]# ceph osd crush remove osd.22100 > device 'osd.22100' does not appear in the crush map > > then I tried to delete osd.71 and mon proccess crushed: > > [root@cc1 ~]# ceph osd crush remove osd.71 > 2016-12-28 17:52:34.459668 7f426a862700 0 monclient: hunting for new mon > > after restart of ceph-mon in ceph osd tree it shows: > > # ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -7 16.89590 root ssd-disks > -11 0 host ssd1 > 598798032 0 osd.598798032 DNE 0 Yikes! > 21940 0 osd.21940 DNE 0 > 71 0 osd.71 DNE 0 > > My question is how to delete this osds without direct editing crushmap > ? It is production system, I can't affort any service interruption :(, > when I try to ceph osd crush remove then ceph-mon crushes.... > > I dumped crushmap, but it took 19G (!!) after decompiling (compiled > file is very small). So, I cleaned this file with perl (it take very > long time), and I have now small txt crushmap, which I edited. But is > there any chance that ceph will still remember somewhere about this > huge numbers for osds ? Is it safe to apply this cleaned crushmap to > cluster ? It sounds like the problem is the OSDMap, not CRUSH per se. Can you attach the output from 'ceph osd dump -f json-pretty'? Do you know how osd.598798032 got created? Or osd.21940 for that matter. OSD ids should be small since they are stored internally by OSDMap as a vector. This is probably why your mon is crashing. sage ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 19:10 ` Sage Weil @ 2016-12-29 20:20 ` Łukasz Chrustek 2016-12-29 20:26 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Łukasz Chrustek @ 2016-12-29 20:20 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel Hi, >> >> # ceph osd tree >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> -7 16.89590 root ssd-disks >> -11 0 host ssd1 >> 598798032 0 osd.598798032 DNE 0 > Yikes! Yes... indeed, I don't like this number too... >> 21940 0 osd.21940 DNE 0 >> 71 0 osd.71 DNE 0 >> >> My question is how to delete this osds without direct editing crushmap >> ? It is production system, I can't affort any service interruption :(, >> when I try to ceph osd crush remove then ceph-mon crushes.... >> >> I dumped crushmap, but it took 19G (!!) after decompiling (compiled >> file is very small). So, I cleaned this file with perl (it take very >> long time), and I have now small txt crushmap, which I edited. But is >> there any chance that ceph will still remember somewhere about this >> huge numbers for osds ? Is it safe to apply this cleaned crushmap to >> cluster ? > It sounds like the problem is the OSDMap, not CRUSH per se. Can you > attach the output from 'ceph osd dump -f json-pretty'? It's quite big so I put it on pastebin: http://pastebin.com/Unkk2Pa7 > Do you know how osd.598798032 got created? Or osd.21940 for that matter. > OSD ids should be small since they are stored internally by OSDMap as a > vector. This is probably why your mon is crashing. [root@cc1 /etc/ceph]# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -7 16.89590 root ssd-intel-s3700 -11 0 host ssd-stor1 69 0 osd.69 down 0 1.00000 70 0 osd.70 down 0 1.00000 71 0 osd.71 down 0 1.00000 This the moment, when it happend: ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done removed item id 69 name 'osd.69' from crush map removed item id 70 name 'osd.70' from crush map here i press ctrl+c 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault after restart of ceph-mon: ]# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -7 16.89590 root ssd-intel-s3700 -11 0 host ssd-stor1 -231707408 0 22100 0 osd.22100 DNE 0 71 0 osd.71 DNE 0 and later: [root@cc1 ~]# ceph osd crush remove osd.22100 device 'osd.22100' does not appear in the crush map [root@cc1 ~]# ceph osd crush remove osd.71 2016-12-28 17:52:34.459668 7f426a862700 0 monclient: hunting for new mon 2016-12-28 17:52:55.238418 7f426a862700 0 monclient: hunting for new mon 2016-12-28 17:52:55.238680 7f4262ebc700 0 -- 192.168.128.1:0/692048545 >> 192.168.128.2:6789/0 pipe(0x7f4254028300 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4254026800).fault and after another restart of ceph-mon: ]# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -7 16.89590 root ssd-intel-s3700 -11 0 host ssd-stor1 598798032 0 osd.598798032 DNE 0 21940 0 osd.21940 DNE 0 71 0 osd.71 DNE 0 -- Regards Luk ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 20:20 ` Łukasz Chrustek @ 2016-12-29 20:26 ` Sage Weil 2016-12-29 20:46 ` Łukasz Chrustek 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2016-12-29 20:26 UTC (permalink / raw) To: Łukasz Chrustek; +Cc: ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 4393 bytes --] On Thu, 29 Dec 2016, Łukasz Chrustek wrote: > Hi, > > > >> > >> # ceph osd tree > >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > >> -7 16.89590 root ssd-disks > >> -11 0 host ssd1 > >> 598798032 0 osd.598798032 DNE 0 > > > Yikes! > > Yes... indeed, I don't like this number too... > > >> 21940 0 osd.21940 DNE 0 > >> 71 0 osd.71 DNE 0 > >> > >> My question is how to delete this osds without direct editing crushmap > >> ? It is production system, I can't affort any service interruption :(, > >> when I try to ceph osd crush remove then ceph-mon crushes.... > >> > >> I dumped crushmap, but it took 19G (!!) after decompiling (compiled > >> file is very small). So, I cleaned this file with perl (it take very > >> long time), and I have now small txt crushmap, which I edited. But is > >> there any chance that ceph will still remember somewhere about this > >> huge numbers for osds ? Is it safe to apply this cleaned crushmap to > >> cluster ? > > > It sounds like the problem is the OSDMap, not CRUSH per se. Can you > > attach the output from 'ceph osd dump -f json-pretty'? > > It's quite big so I put it on pastebin: > > http://pastebin.com/Unkk2Pa7 > > > Do you know how osd.598798032 got created? Or osd.21940 for that matter. > > OSD ids should be small since they are stored internally by OSDMap as a > > vector. This is probably why your mon is crashing. > > [root@cc1 /etc/ceph]# ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -7 16.89590 root ssd-intel-s3700 > -11 0 host ssd-stor1 > 69 0 osd.69 down 0 1.00000 > 70 0 osd.70 down 0 1.00000 > 71 0 osd.71 down 0 1.00000 > > > This the moment, when it happend: > ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done > removed item id 69 name 'osd.69' from crush map > > > removed item id 70 name 'osd.70' from crush map > > here i press ctrl+c > > 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon > 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault > 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault > 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault What version is this? Can you attach the crush map too? (ceph osd crush dump -f json-pretty) Thanks! sage > after restart of ceph-mon: > > ]# ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -7 16.89590 root ssd-intel-s3700 > -11 0 host ssd-stor1 > -231707408 0 > 22100 0 osd.22100 DNE 0 > 71 0 osd.71 DNE 0 > > and later: > > [root@cc1 ~]# ceph osd crush remove osd.22100 > device 'osd.22100' does not appear in the crush map > [root@cc1 ~]# ceph osd crush remove osd.71 > 2016-12-28 17:52:34.459668 7f426a862700 0 monclient: hunting for new mon > 2016-12-28 17:52:55.238418 7f426a862700 0 monclient: hunting for new mon > 2016-12-28 17:52:55.238680 7f4262ebc700 0 -- 192.168.128.1:0/692048545 >> 192.168.128.2:6789/0 pipe(0x7f4254028300 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4254026800).fault > > and after another restart of ceph-mon: > > ]# ceph osd tree > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > -7 16.89590 root ssd-intel-s3700 > -11 0 host ssd-stor1 > 598798032 0 osd.598798032 DNE 0 > 21940 0 osd.21940 DNE 0 > 71 0 osd.71 DNE 0 > > > > > -- > Regards > Luk > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 20:26 ` Sage Weil @ 2016-12-29 20:46 ` Łukasz Chrustek 2016-12-29 20:49 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Łukasz Chrustek @ 2016-12-29 20:46 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel Cześć, > On Thu, 29 Dec 2016, Łukasz Chrustek wrote: >> Hi, >> >> >> >> >> >> # ceph osd tree >> >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> >> -7 16.89590 root ssd-disks >> >> -11 0 host ssd1 >> >> 598798032 0 osd.598798032 DNE 0 >> >> > Yikes! >> >> Yes... indeed, I don't like this number too... >> >> >> 21940 0 osd.21940 DNE 0 >> >> 71 0 osd.71 DNE 0 >> >> >> >> My question is how to delete this osds without direct editing crushmap >> >> ? It is production system, I can't affort any service interruption :(, >> >> when I try to ceph osd crush remove then ceph-mon crushes.... >> >> >> >> I dumped crushmap, but it took 19G (!!) after decompiling (compiled >> >> file is very small). So, I cleaned this file with perl (it take very >> >> long time), and I have now small txt crushmap, which I edited. But is >> >> there any chance that ceph will still remember somewhere about this >> >> huge numbers for osds ? Is it safe to apply this cleaned crushmap to >> >> cluster ? >> >> > It sounds like the problem is the OSDMap, not CRUSH per se. Can you >> > attach the output from 'ceph osd dump -f json-pretty'? >> >> It's quite big so I put it on pastebin: >> >> http://pastebin.com/Unkk2Pa7 >> >> > Do you know how osd.598798032 got created? Or osd.21940 for that matter. >> > OSD ids should be small since they are stored internally by OSDMap as a >> > vector. This is probably why your mon is crashing. >> >> [root@cc1 /etc/ceph]# ceph osd tree >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> -7 16.89590 root ssd-intel-s3700 >> -11 0 host ssd-stor1 >> 69 0 osd.69 down 0 1.00000 >> 70 0 osd.70 down 0 1.00000 >> 71 0 osd.71 down 0 1.00000 >> >> >> This the moment, when it happend: >> ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done >> removed item id 69 name 'osd.69' from crush map >> >> >> removed item id 70 name 'osd.70' from crush map >> >> here i press ctrl+c >> >> 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon >> 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault >> 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault >> 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault > What version is this? infernalis > Can you attach the crush map too? (ceph osd crush dump -f json-pretty) I can't - ceph-mons are crushing on diffrent ceph-mon hosts: ]# ceph osd crush dump -f json-pretty 2016-12-29 21:42:47.172097 7f0c36061700 0 monclient: hunting for new mon ^CTraceback (most recent call last): File "/usr/bin/ceph", line 948, in <module> retval = main() File "/usr/bin/ceph", line 881, in main sigdict, inbuf, verbose) File "/usr/bin/ceph", line 479, in new_style_command inbuf=inbuf) File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1291, in json_command raise RuntimeError('"{0}": exception {1}'.format(argdict, e)) RuntimeError: "{'prefix': u'osd crush dump', 'format': 'json-pretty'}": exception "['{"prefix": "osd crush dump", "format": "json-pretty"}']": exception 'int' object is not iterable -- Pozdrowienia, Łukasz Chrustek ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 20:46 ` Łukasz Chrustek @ 2016-12-29 20:49 ` Sage Weil 2016-12-29 20:55 ` Łukasz Chrustek 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2016-12-29 20:49 UTC (permalink / raw) To: Łukasz Chrustek; +Cc: ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 3336 bytes --] On Thu, 29 Dec 2016, Łukasz Chrustek wrote: > Cześć, > > > On Thu, 29 Dec 2016, Łukasz Chrustek wrote: > >> Hi, > >> > >> > >> >> > >> >> # ceph osd tree > >> >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > >> >> -7 16.89590 root ssd-disks > >> >> -11 0 host ssd1 > >> >> 598798032 0 osd.598798032 DNE 0 > >> > >> > Yikes! > >> > >> Yes... indeed, I don't like this number too... > >> > >> >> 21940 0 osd.21940 DNE 0 > >> >> 71 0 osd.71 DNE 0 > >> >> > >> >> My question is how to delete this osds without direct editing crushmap > >> >> ? It is production system, I can't affort any service interruption :(, > >> >> when I try to ceph osd crush remove then ceph-mon crushes.... > >> >> > >> >> I dumped crushmap, but it took 19G (!!) after decompiling (compiled > >> >> file is very small). So, I cleaned this file with perl (it take very > >> >> long time), and I have now small txt crushmap, which I edited. But is > >> >> there any chance that ceph will still remember somewhere about this > >> >> huge numbers for osds ? Is it safe to apply this cleaned crushmap to > >> >> cluster ? > >> > >> > It sounds like the problem is the OSDMap, not CRUSH per se. Can you > >> > attach the output from 'ceph osd dump -f json-pretty'? > >> > >> It's quite big so I put it on pastebin: > >> > >> http://pastebin.com/Unkk2Pa7 > >> > >> > Do you know how osd.598798032 got created? Or osd.21940 for that matter. > >> > OSD ids should be small since they are stored internally by OSDMap as a > >> > vector. This is probably why your mon is crashing. > >> > >> [root@cc1 /etc/ceph]# ceph osd tree > >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > >> -7 16.89590 root ssd-intel-s3700 > >> -11 0 host ssd-stor1 > >> 69 0 osd.69 down 0 1.00000 > >> 70 0 osd.70 down 0 1.00000 > >> 71 0 osd.71 down 0 1.00000 > >> > >> > >> This the moment, when it happend: > >> ]# for i in `seq 69 71`;do ceph osd crush remove osd.$i;done > >> removed item id 69 name 'osd.69' from crush map > >> > >> > >> removed item id 70 name 'osd.70' from crush map > >> > >> here i press ctrl+c > >> > >> 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon > >> 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault > >> 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault > >> 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault > > > What version is this? > > infernalis > > > Can you attach the crush map too? (ceph osd crush dump -f json-pretty) > > I can't - ceph-mons are crushing on diffrent ceph-mon hosts: > > ]# ceph osd crush dump -f json-pretty Hmm, in that case, 'ceph osd getcrushmap -o cm' and post that somewhere? sage ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 20:49 ` Sage Weil @ 2016-12-29 20:55 ` Łukasz Chrustek 2016-12-29 21:41 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Łukasz Chrustek @ 2016-12-29 20:55 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel [-- Attachment #1: Type: text/plain, Size: 1045 bytes --] Cześć, >> >> >> >> here i press ctrl+c >> >> >> >> 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon >> >> 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault >> >> 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault >> >> 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault >> >> > What version is this? >> >> infernalis >> >> > Can you attach the crush map too? (ceph osd crush dump -f json-pretty) >> >> I can't - ceph-mons are crushing on diffrent ceph-mon hosts: >> >> ]# ceph osd crush dump -f json-pretty > Hmm, in that case, 'ceph osd getcrushmap -o cm' and post that somewhere? In attachment. > sage -- Regards Luk [-- Attachment #2: cm --] [-- Type: application/octet-stream, Size: 3559 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 20:55 ` Łukasz Chrustek @ 2016-12-29 21:41 ` Sage Weil 2016-12-29 22:02 ` Łukasz Chrustek 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2016-12-29 21:41 UTC (permalink / raw) To: Łukasz Chrustek; +Cc: ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 1660 bytes --] On Thu, 29 Dec 2016, Łukasz Chrustek wrote: > Cześć, > > > >> >> > >> >> here i press ctrl+c > >> >> > >> >> 2016-12-28 17:38:10.055239 7f4576d7a700 0 monclient: hunting for new mon > >> >> 2016-12-28 17:38:10.055582 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f456c023190 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f456c024470).fault > >> >> 2016-12-28 17:38:30.550622 7f4574233700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.1:6789/0 pipe(0x7f45600008c0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4560001df0).fault > >> >> 2016-12-28 17:38:54.551031 7f4574474700 0 -- 192.168.128.1:0/1201679761 >> 192.168.128.2:6789/0 pipe(0x7f45600046c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f45600042b0).fault > >> > >> > What version is this? > >> > >> infernalis > >> > >> > Can you attach the crush map too? (ceph osd crush dump -f json-pretty) > >> > >> I can't - ceph-mons are crushing on diffrent ceph-mon hosts: > >> > >> ]# ceph osd crush dump -f json-pretty > > > Hmm, in that case, 'ceph osd getcrushmap -o cm' and post that somewhere? > > In attachment. The encoded tree bucket -11 had bad values. I don't really trust the tree bucket code in crush... it's not well tested (and is a poor balance computation and efficiency anyway). We should probably try to remove tree entirely. I've attached a fixed map that you can inject with ceph osd setcrushmap -i <filename> Bucket -11 is now empty; not sure what was supposed to be in it. I suggest switching all of your tree buckets over to straw2 as soon as possible. Note that this will result in some rebalancing. You could do it one bucket a time if that's concerning. sage [-- Attachment #2: Type: APPLICATION/octet-stream, Size: 3627 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 21:41 ` Sage Weil @ 2016-12-29 22:02 ` Łukasz Chrustek 2016-12-29 22:18 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Łukasz Chrustek @ 2016-12-29 22:02 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel Hi, Thank You very much for analize and the file ! I had similar :) but wasn't sure if it wan't distroy something in cluster. > The encoded tree bucket -11 had bad values. I don't really trust the tree > bucket code in crush... it's not well tested (and is a poor balance > computation and efficiency anyway). We should probably try to remove tree > entirely. > I've attached a fixed map that you can inject with > ceph osd setcrushmap -i <filename> Now it works, and also ceph osd crush dump -f json-pretty runs OK. > Bucket -11 is now empty; not sure what was supposed to be in it. this server will be reinstalled, there where three osds. > I suggest switching all of your tree buckets over to straw2 as soon as > possible. Note that this will result in some rebalancing. You could do > it one bucket a time if that's concerning. OK, changing alg to straw2 will rebalance ale PGs on all nodes ? -- Regards, Luk ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 22:02 ` Łukasz Chrustek @ 2016-12-29 22:18 ` Sage Weil 2016-12-29 23:13 ` Brad Hubbard 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2016-12-29 22:18 UTC (permalink / raw) To: Łukasz Chrustek; +Cc: ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 1366 bytes --] On Thu, 29 Dec 2016, Łukasz Chrustek wrote: > Hi, > > Thank You very much for analize and the file ! I had similar :) but > wasn't sure if it wan't distroy something in cluster. > > > The encoded tree bucket -11 had bad values. I don't really trust the tree > > bucket code in crush... it's not well tested (and is a poor balance > > computation and efficiency anyway). We should probably try to remove tree > > entirely. > > > I've attached a fixed map that you can inject with > > > ceph osd setcrushmap -i <filename> > > Now it works, and also ceph osd crush dump -f json-pretty runs OK. Great news! > > Bucket -11 is now empty; not sure what was supposed to be in it. > > this server will be reinstalled, there where three osds. > > > I suggest switching all of your tree buckets over to straw2 as soon as > > possible. Note that this will result in some rebalancing. You could do > > it one bucket a time if that's concerning. > > OK, changing alg to straw2 will rebalance ale PGs on all nodes ? For any bucket you change from tree -> straw2, you'll see PGs shuffle between the children of that bucket. So for hosts, you'll see data move between the disks. And fixing ssd-intel-s3700 will shuffle data between hosts. I'd also switch the straw buckets to straw2, although that will move a comparatively small amount of data. sage ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 22:18 ` Sage Weil @ 2016-12-29 23:13 ` Brad Hubbard 2016-12-29 23:33 ` Łukasz Chrustek 0 siblings, 1 reply; 12+ messages in thread From: Brad Hubbard @ 2016-12-29 23:13 UTC (permalink / raw) To: Sage Weil; +Cc: Łukasz Chrustek, ceph-devel Have you thought through your use of '"min_size": 1' on many of these pools? See recent discussions on this mailing list on the subject for background. Just FYI. On Fri, Dec 30, 2016 at 8:18 AM, Sage Weil <sage@newdream.net> wrote: > On Thu, 29 Dec 2016, Łukasz Chrustek wrote: >> Hi, >> >> Thank You very much for analize and the file ! I had similar :) but >> wasn't sure if it wan't distroy something in cluster. >> >> > The encoded tree bucket -11 had bad values. I don't really trust the tree >> > bucket code in crush... it's not well tested (and is a poor balance >> > computation and efficiency anyway). We should probably try to remove tree >> > entirely. >> >> > I've attached a fixed map that you can inject with >> >> > ceph osd setcrushmap -i <filename> >> >> Now it works, and also ceph osd crush dump -f json-pretty runs OK. > > Great news! > >> > Bucket -11 is now empty; not sure what was supposed to be in it. >> >> this server will be reinstalled, there where three osds. >> >> > I suggest switching all of your tree buckets over to straw2 as soon as >> > possible. Note that this will result in some rebalancing. You could do >> > it one bucket a time if that's concerning. >> >> OK, changing alg to straw2 will rebalance ale PGs on all nodes ? > > For any bucket you change from tree -> straw2, you'll see PGs shuffle > between the children of that bucket. So for hosts, you'll see data move > between the disks. And fixing ssd-intel-s3700 will shuffle data between > hosts. > > I'd also switch the straw buckets to straw2, although that will move a > comparatively small amount of data. > > sage -- Cheers, Brad ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: problem with removing osd 2016-12-29 23:13 ` Brad Hubbard @ 2016-12-29 23:33 ` Łukasz Chrustek 0 siblings, 0 replies; 12+ messages in thread From: Łukasz Chrustek @ 2016-12-29 23:33 UTC (permalink / raw) To: Brad Hubbard, Sage Weil; +Cc: ceph-devel Hi, Thank You for suggestion - I will read about it. Regards Luk > Have you thought through your use of '"min_size": 1' on many of these pools? > See recent discussions on this mailing list on the subject for background. > Just FYI. > On Fri, Dec 30, 2016 at 8:18 AM, Sage Weil <sage@newdream.net> wrote: >> On Thu, 29 Dec 2016, Łukasz Chrustek wrote: >>> Hi, >>> >>> Thank You very much for analize and the file ! I had similar :) but >>> wasn't sure if it wan't distroy something in cluster. >>> >>> > The encoded tree bucket -11 had bad values. I don't really trust the tree >>> > bucket code in crush... it's not well tested (and is a poor balance >>> > computation and efficiency anyway). We should probably try to remove tree >>> > entirely. >>> >>> > I've attached a fixed map that you can inject with >>> >>> > ceph osd setcrushmap -i <filename> >>> >>> Now it works, and also ceph osd crush dump -f json-pretty runs OK. >> >> Great news! >> >>> > Bucket -11 is now empty; not sure what was supposed to be in it. >>> >>> this server will be reinstalled, there where three osds. >>> >>> > I suggest switching all of your tree buckets over to straw2 as soon as >>> > possible. Note that this will result in some rebalancing. You could do >>> > it one bucket a time if that's concerning. >>> >>> OK, changing alg to straw2 will rebalance ale PGs on all nodes ? >> >> For any bucket you change from tree -> straw2, you'll see PGs shuffle >> between the children of that bucket. So for hosts, you'll see data move >> between the disks. And fixing ssd-intel-s3700 will shuffle data between >> hosts. >> >> I'd also switch the straw buckets to straw2, although that will move a >> comparatively small amount of data. >> >> sage -- Pozdrowienia, Łukasz Chrustek ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-12-29 23:33 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-12-29 15:07 problem with removing osd Łukasz Chrustek 2016-12-29 19:10 ` Sage Weil 2016-12-29 20:20 ` Łukasz Chrustek 2016-12-29 20:26 ` Sage Weil 2016-12-29 20:46 ` Łukasz Chrustek 2016-12-29 20:49 ` Sage Weil 2016-12-29 20:55 ` Łukasz Chrustek 2016-12-29 21:41 ` Sage Weil 2016-12-29 22:02 ` Łukasz Chrustek 2016-12-29 22:18 ` Sage Weil 2016-12-29 23:13 ` Brad Hubbard 2016-12-29 23:33 ` Łukasz Chrustek
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.