* ceph cluster hangs when rebooting one node
@ 2012-11-12 15:04 Stefan Priebe - Profihost AG
2012-11-12 15:11 ` Sage Weil
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-12 15:04 UTC (permalink / raw)
To: ceph-devel@vger.kernel.org
Hello list,
i was checking what happens if i reboot a ceph node.
Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is
possible.
ceph -w:
Looks like this:
2012-11-12 16:03:58.191106 mon.0 [INF] pgmap v19013: 7032 pgs: 7032
active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
2012-11-12 16:04:08.365557 mon.0 [INF] mon.a calling new monitor election
2012-11-12 16:04:13.422682 mon.0 [INF] mon.a@0 won leader election with
quorum 0,2
2012-11-12 16:04:13.708045 mon.0 [INF] pgmap v19014: 7032 pgs: 7032
active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
2012-11-12 16:04:13.708059 mon.0 [INF] mdsmap e1: 0/0/1 up
2012-11-12 16:04:13.708070 mon.0 [INF] osdmap e4582: 20 osds: 20 up, 20 in
2012-11-12 16:04:08.242688 mon.2 [INF] mon.c calling new monitor election
2012-11-12 16:04:13.708089 mon.0 [INF] monmap e1: 3 mons at
{a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0}
2012-11-12 16:04:14.070593 mon.0 [INF] pgmap v19015: 7032 pgs: 7032
active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
2012-11-12 16:04:15.283954 mon.0 [INF] pgmap v19016: 7032 pgs: 7032
active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
2012-11-12 16:04:18.506812 mon.0 [INF] osd.21 10.255.0.101:6800/5049
failed (3 reports from 3 peers after 20.339769 >= grace 20.000000)
2012-11-12 16:04:18.890003 mon.0 [INF] osdmap e4583: 20 osds: 19 up, 20 in
2012-11-12 16:04:19.137936 mon.0 [INF] pgmap v19017: 7032 pgs: 6720
active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294
GB / 4469 GB avail
2012-11-12 16:04:20.024595 mon.0 [INF] osdmap e4584: 20 osds: 19 up, 20 in
2012-11-12 16:04:20.330149 mon.0 [INF] pgmap v19018: 7032 pgs: 6720
active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294
GB / 4469 GB avail
2012-11-12 16:04:21.535471 mon.0 [INF] pgmap v19019: 7032 pgs: 6720
active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294
GB / 4469 GB avail
2012-11-12 16:04:24.181292 mon.0 [INF] osd.22 10.255.0.101:6803/5153
failed (3 reports from 3 peers after 23.013550 >= grace 20.000000)
2012-11-12 16:04:24.182208 mon.0 [INF] osd.23 10.255.0.101:6806/5276
failed (3 reports from 3 peers after 21.000834 >= grace 20.000000)
2012-11-12 16:04:24.671373 mon.0 [INF] pgmap v19020: 7032 pgs: 6637
active+clean, 208 stale+active+clean, 187 incomplete; 91615 MB data, 174
GB used, 4295 GB / 4469 GB avail
2012-11-12 16:04:24.829022 mon.0 [INF] osdmap e4585: 20 osds: 17 up, 20 in
2012-11-12 16:04:24.870969 mon.0 [INF] osd.24 10.255.0.101:6809/5397
failed (3 reports from 3 peers after 20.688672 >= grace 20.000000)
2012-11-12 16:04:25.522333 mon.0 [INF] pgmap v19021: 7032 pgs: 5912
active+clean, 933 stale+active+clean, 187 incomplete; 91615 MB data, 174
GB used, 4295 GB / 4469 GB avail
2012-11-12 16:04:25.596927 mon.0 [INF] osd.24 10.255.0.101:6809/5397
failed (3 reports from 3 peers after 21.708444 >= grace 20.000000)
2012-11-12 16:04:26.077545 mon.0 [INF] osdmap e4586: 20 osds: 16 up, 20 in
2012-11-12 16:04:26.606475 mon.0 [INF] pgmap v19022: 7032 pgs: 5394
active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data,
173 GB used, 4296 GB / 4469 GB avail
2012-11-12 16:04:27.162034 mon.0 [INF] osdmap e4587: 20 osds: 16 up, 20 in
2012-11-12 16:04:27.656974 mon.0 [INF] pgmap v19023: 7032 pgs: 5394
active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data,
173 GB used, 4296 GB / 4469 GB avail
2012-11-12 16:04:30.229958 mon.0 [INF] pgmap v19024: 7032 pgs: 5394
active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data,
172 GB used, 4296 GB / 4469 GB avail
2012-11-12 16:04:31.411989 mon.0 [INF] pgmap v19025: 7032 pgs: 5394
active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data,
172 GB used, 4296 GB / 4469 GB avail
2012-11-12 16:04:32.617576 mon.0 [INF] pgmap v19026: 7032 pgs: 4660
active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB /
4469 GB avail
2012-11-12 16:04:35.172861 mon.0 [INF] pgmap v19027: 7032 pgs: 4660
active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB /
4469 GB avail
2012-11-12 16:04:30.505872 osd.53 [WRN] 6 slow requests, 6 included
below; oldest blocked for > 30.247691 secs
2012-11-12 16:04:30.505875 osd.53 [WRN] slow request 30.247691 seconds
old, received at 2012-11-12 16:04:00.258118:
osd_op(client.131626.0:771962 rb.0.107a.734602d5.000000000bce [write
2478080~4096] 3.562a9efc) v4 currently reached pg
2012-11-12 16:04:30.505879 osd.53 [WRN] slow request 30.238016 seconds
old, received at 2012-11-12 16:04:00.267793:
osd_op(client.131626.0:772116 rb.0.107a.734602d5.000000001608 [write
262144~4096] 3.a47890e) v4 currently reached pg
2012-11-12 16:04:30.505881 osd.53 [WRN] slow request 30.236572 seconds
old, received at 2012-11-12 16:04:00.269237:
osd_op(client.131626.0:772141 rb.0.107a.734602d5.000000001777 [write
798720~4096] 3.547bc855) v4 currently reached pg
2012-11-12 16:04:30.505883 osd.53 [WRN] slow request 30.227850 seconds
old, received at 2012-11-12 16:04:00.277959:
osd_op(client.131626.0:772283 rb.0.107a.734602d5.0000000000a6 [write
2379776~4096] 3.5d0f2510) v4 currently reached pg
2012-11-12 16:04:30.505884 osd.53 [WRN] slow request 30.227499 seconds
old, received at 2012-11-12 16:04:00.278310:
osd_op(client.131626.0:772289 rb.0.107a.734602d5.0000000000d0 [write
3379200~4096] 3.b031884f) v4 currently reached pg
2012-11-12 16:04:30.819063 osd.52 [WRN] 6 slow requests, 6 included
below; oldest blocked for > 30.578003 secs
2012-11-12 16:04:30.819069 osd.52 [WRN] slow request 30.578003 seconds
old, received at 2012-11-12 16:04:00.240978:
osd_op(client.131626.0:771697 rb.0.107a.734602d5.000000001916 [write
3076096~4096] 3.627cbcb1) v4 currently reached pg
2012-11-12 16:04:30.819076 osd.52 [WRN] slow request 30.546967 seconds
old, received at 2012-11-12 16:04:00.272014:
osd_op(client.131626.0:772187 rb.0.107a.734602d5.000000001974 [write
1675264~4096] 3.ba912483) v4 currently reached pg
2012-11-12 16:04:30.819078 osd.52 [WRN] slow request 30.544082 seconds
old, received at 2012-11-12 16:04:00.274899:
osd_op(client.131626.0:772235 rb.0.107a.734602d5.000000001bfd [write
3686400~4096] 3.29b75f52) v4 currently reached pg
2012-11-12 16:04:30.819080 osd.52 [WRN] slow request 30.496902 seconds
old, received at 2012-11-12 16:04:00.322079:
osd_op(client.131626.0:772944 rb.0.107a.734602d5.000000000bbb [write
266240~4096] 3.5db27880) v4 currently reached pg
2012-11-12 16:04:30.819081 osd.52 [WRN] slow request 30.470500 seconds
old, received at 2012-11-12 16:04:00.348481:
osd_op(client.131626.0:773397 rb.0.107a.734602d5.000000000bbb [write
4145152~4096] 3.5db27880) v4 currently reached pg
2012-11-12 16:04:31.202553 osd.51 [WRN] 6 slow requests, 6 included
below; oldest blocked for > 30.932114 secs
2012-11-12 16:04:31.203126 osd.51 [WRN] slow request 30.932114 seconds
old, received at 2012-11-12 16:04:00.270383:
osd_op(client.131626.0:772159 rb.0.107a.734602d5.000000001826 [write
3842048~4096] 3.d489eb11) v4 currently reached pg
2012-11-12 16:04:31.203130 osd.51 [WRN] slow request 30.902220 seconds
old, received at 2012-11-12 16:04:00.300277:
osd_op(client.131626.0:772552 rb.0.107a.734602d5.000000000fd9 [write
2990080~4096] 3.e64d168c) v4 currently reached pg
2012-11-12 16:04:31.203132 osd.51 [WRN] slow request 30.895459 seconds
old, received at 2012-11-12 16:04:00.307038:
osd_op(client.131626.0:772670 rb.0.107a.734602d5.00000000177f [write
1028096~4096] 3.dad40d42) v4 currently reached pg
2012-11-12 16:04:31.203135 osd.51 [WRN] slow request 30.891418 seconds
old, received at 2012-11-12 16:04:00.311079:
osd_op(client.131626.0:772730 rb.0.107a.734602d5.000000001ac6 [write
495616~4096] 3.27fd6b11) v4 currently reached pg
2012-11-12 16:04:31.203136 osd.51 [WRN] slow request 30.845134 seconds
old, received at 2012-11-12 16:04:00.357363:
osd_op(client.131626.0:773553 rb.0.107a.734602d5.000000001688 [write
3125248~4096] 3.c83fad42) v4 currently reached pg
Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: ceph cluster hangs when rebooting one node
2012-11-12 15:04 ceph cluster hangs when rebooting one node Stefan Priebe - Profihost AG
@ 2012-11-12 15:11 ` Sage Weil
2012-11-12 15:24 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 5+ messages in thread
From: Sage Weil @ 2012-11-12 15:11 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-devel@vger.kernel.org
On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote:
> Hello list,
>
> i was checking what happens if i reboot a ceph node.
>
> Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is
> possible.
If you are using the current master, the new 'min_size' may be biting you;
ceph osd dump | grep ^pool and see if you see min_size for your pools.
You can change that back to the norma behavior with
ceph osd pool set <poolname> min_size 1
sage
>
> ceph -w:
> Looks like this:
> 2012-11-12 16:03:58.191106 mon.0 [INF] pgmap v19013: 7032 pgs: 7032
> active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
> 2012-11-12 16:04:08.365557 mon.0 [INF] mon.a calling new monitor election
> 2012-11-12 16:04:13.422682 mon.0 [INF] mon.a@0 won leader election with quorum
> 0,2
> 2012-11-12 16:04:13.708045 mon.0 [INF] pgmap v19014: 7032 pgs: 7032
> active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
> 2012-11-12 16:04:13.708059 mon.0 [INF] mdsmap e1: 0/0/1 up
> 2012-11-12 16:04:13.708070 mon.0 [INF] osdmap e4582: 20 osds: 20 up, 20 in
> 2012-11-12 16:04:08.242688 mon.2 [INF] mon.c calling new monitor election
> 2012-11-12 16:04:13.708089 mon.0 [INF] monmap e1: 3 mons at
> {a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0}
> 2012-11-12 16:04:14.070593 mon.0 [INF] pgmap v19015: 7032 pgs: 7032
> active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
> 2012-11-12 16:04:15.283954 mon.0 [INF] pgmap v19016: 7032 pgs: 7032
> active+clean; 91615 MB data, 174 GB used, 4294 GB / 4469 GB avail
> 2012-11-12 16:04:18.506812 mon.0 [INF] osd.21 10.255.0.101:6800/5049 failed (3
> reports from 3 peers after 20.339769 >= grace 20.000000)
> 2012-11-12 16:04:18.890003 mon.0 [INF] osdmap e4583: 20 osds: 19 up, 20 in
> 2012-11-12 16:04:19.137936 mon.0 [INF] pgmap v19017: 7032 pgs: 6720
> active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB /
> 4469 GB avail
> 2012-11-12 16:04:20.024595 mon.0 [INF] osdmap e4584: 20 osds: 19 up, 20 in
> 2012-11-12 16:04:20.330149 mon.0 [INF] pgmap v19018: 7032 pgs: 6720
> active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB /
> 4469 GB avail
> 2012-11-12 16:04:21.535471 mon.0 [INF] pgmap v19019: 7032 pgs: 6720
> active+clean, 312 stale+active+clean; 91615 MB data, 174 GB used, 4294 GB /
> 4469 GB avail
> 2012-11-12 16:04:24.181292 mon.0 [INF] osd.22 10.255.0.101:6803/5153 failed (3
> reports from 3 peers after 23.013550 >= grace 20.000000)
> 2012-11-12 16:04:24.182208 mon.0 [INF] osd.23 10.255.0.101:6806/5276 failed (3
> reports from 3 peers after 21.000834 >= grace 20.000000)
> 2012-11-12 16:04:24.671373 mon.0 [INF] pgmap v19020: 7032 pgs: 6637
> active+clean, 208 stale+active+clean, 187 incomplete; 91615 MB data, 174 GB
> used, 4295 GB / 4469 GB avail
> 2012-11-12 16:04:24.829022 mon.0 [INF] osdmap e4585: 20 osds: 17 up, 20 in
> 2012-11-12 16:04:24.870969 mon.0 [INF] osd.24 10.255.0.101:6809/5397 failed (3
> reports from 3 peers after 20.688672 >= grace 20.000000)
> 2012-11-12 16:04:25.522333 mon.0 [INF] pgmap v19021: 7032 pgs: 5912
> active+clean, 933 stale+active+clean, 187 incomplete; 91615 MB data, 174 GB
> used, 4295 GB / 4469 GB avail
> 2012-11-12 16:04:25.596927 mon.0 [INF] osd.24 10.255.0.101:6809/5397 failed (3
> reports from 3 peers after 21.708444 >= grace 20.000000)
> 2012-11-12 16:04:26.077545 mon.0 [INF] osdmap e4586: 20 osds: 16 up, 20 in
> 2012-11-12 16:04:26.606475 mon.0 [INF] pgmap v19022: 7032 pgs: 5394
> active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 173 GB
> used, 4296 GB / 4469 GB avail
> 2012-11-12 16:04:27.162034 mon.0 [INF] osdmap e4587: 20 osds: 16 up, 20 in
> 2012-11-12 16:04:27.656974 mon.0 [INF] pgmap v19023: 7032 pgs: 5394
> active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 173 GB
> used, 4296 GB / 4469 GB avail
> 2012-11-12 16:04:30.229958 mon.0 [INF] pgmap v19024: 7032 pgs: 5394
> active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 172 GB
> used, 4296 GB / 4469 GB avail
> 2012-11-12 16:04:31.411989 mon.0 [INF] pgmap v19025: 7032 pgs: 5394
> active+clean, 1094 stale+active+clean, 544 incomplete; 91615 MB data, 172 GB
> used, 4296 GB / 4469 GB avail
> 2012-11-12 16:04:32.617576 mon.0 [INF] pgmap v19026: 7032 pgs: 4660
> active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB / 4469 GB
> avail
> 2012-11-12 16:04:35.172861 mon.0 [INF] pgmap v19027: 7032 pgs: 4660
> active+clean, 2372 incomplete; 91615 MB data, 171 GB used, 4298 GB / 4469 GB
> avail
> 2012-11-12 16:04:30.505872 osd.53 [WRN] 6 slow requests, 6 included below;
> oldest blocked for > 30.247691 secs
> 2012-11-12 16:04:30.505875 osd.53 [WRN] slow request 30.247691 seconds old,
> received at 2012-11-12 16:04:00.258118: osd_op(client.131626.0:771962
> rb.0.107a.734602d5.000000000bce [write 2478080~4096] 3.562a9efc) v4 currently
> reached pg
> 2012-11-12 16:04:30.505879 osd.53 [WRN] slow request 30.238016 seconds old,
> received at 2012-11-12 16:04:00.267793: osd_op(client.131626.0:772116
> rb.0.107a.734602d5.000000001608 [write 262144~4096] 3.a47890e) v4 currently
> reached pg
> 2012-11-12 16:04:30.505881 osd.53 [WRN] slow request 30.236572 seconds old,
> received at 2012-11-12 16:04:00.269237: osd_op(client.131626.0:772141
> rb.0.107a.734602d5.000000001777 [write 798720~4096] 3.547bc855) v4 currently
> reached pg
> 2012-11-12 16:04:30.505883 osd.53 [WRN] slow request 30.227850 seconds old,
> received at 2012-11-12 16:04:00.277959: osd_op(client.131626.0:772283
> rb.0.107a.734602d5.0000000000a6 [write 2379776~4096] 3.5d0f2510) v4 currently
> reached pg
> 2012-11-12 16:04:30.505884 osd.53 [WRN] slow request 30.227499 seconds old,
> received at 2012-11-12 16:04:00.278310: osd_op(client.131626.0:772289
> rb.0.107a.734602d5.0000000000d0 [write 3379200~4096] 3.b031884f) v4 currently
> reached pg
> 2012-11-12 16:04:30.819063 osd.52 [WRN] 6 slow requests, 6 included below;
> oldest blocked for > 30.578003 secs
> 2012-11-12 16:04:30.819069 osd.52 [WRN] slow request 30.578003 seconds old,
> received at 2012-11-12 16:04:00.240978: osd_op(client.131626.0:771697
> rb.0.107a.734602d5.000000001916 [write 3076096~4096] 3.627cbcb1) v4 currently
> reached pg
> 2012-11-12 16:04:30.819076 osd.52 [WRN] slow request 30.546967 seconds old,
> received at 2012-11-12 16:04:00.272014: osd_op(client.131626.0:772187
> rb.0.107a.734602d5.000000001974 [write 1675264~4096] 3.ba912483) v4 currently
> reached pg
> 2012-11-12 16:04:30.819078 osd.52 [WRN] slow request 30.544082 seconds old,
> received at 2012-11-12 16:04:00.274899: osd_op(client.131626.0:772235
> rb.0.107a.734602d5.000000001bfd [write 3686400~4096] 3.29b75f52) v4 currently
> reached pg
> 2012-11-12 16:04:30.819080 osd.52 [WRN] slow request 30.496902 seconds old,
> received at 2012-11-12 16:04:00.322079: osd_op(client.131626.0:772944
> rb.0.107a.734602d5.000000000bbb [write 266240~4096] 3.5db27880) v4 currently
> reached pg
> 2012-11-12 16:04:30.819081 osd.52 [WRN] slow request 30.470500 seconds old,
> received at 2012-11-12 16:04:00.348481: osd_op(client.131626.0:773397
> rb.0.107a.734602d5.000000000bbb [write 4145152~4096] 3.5db27880) v4 currently
> reached pg
> 2012-11-12 16:04:31.202553 osd.51 [WRN] 6 slow requests, 6 included below;
> oldest blocked for > 30.932114 secs
> 2012-11-12 16:04:31.203126 osd.51 [WRN] slow request 30.932114 seconds old,
> received at 2012-11-12 16:04:00.270383: osd_op(client.131626.0:772159
> rb.0.107a.734602d5.000000001826 [write 3842048~4096] 3.d489eb11) v4 currently
> reached pg
> 2012-11-12 16:04:31.203130 osd.51 [WRN] slow request 30.902220 seconds old,
> received at 2012-11-12 16:04:00.300277: osd_op(client.131626.0:772552
> rb.0.107a.734602d5.000000000fd9 [write 2990080~4096] 3.e64d168c) v4 currently
> reached pg
> 2012-11-12 16:04:31.203132 osd.51 [WRN] slow request 30.895459 seconds old,
> received at 2012-11-12 16:04:00.307038: osd_op(client.131626.0:772670
> rb.0.107a.734602d5.00000000177f [write 1028096~4096] 3.dad40d42) v4 currently
> reached pg
> 2012-11-12 16:04:31.203135 osd.51 [WRN] slow request 30.891418 seconds old,
> received at 2012-11-12 16:04:00.311079: osd_op(client.131626.0:772730
> rb.0.107a.734602d5.000000001ac6 [write 495616~4096] 3.27fd6b11) v4 currently
> reached pg
> 2012-11-12 16:04:31.203136 osd.51 [WRN] slow request 30.845134 seconds old,
> received at 2012-11-12 16:04:00.357363: osd_op(client.131626.0:773553
> rb.0.107a.734602d5.000000001688 [write 3125248~4096] 3.c83fad42) v4 currently
> reached pg
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: ceph cluster hangs when rebooting one node
2012-11-12 15:11 ` Sage Weil
@ 2012-11-12 15:24 ` Stefan Priebe - Profihost AG
2012-11-14 9:50 ` Aleksey Samarin
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-12 15:24 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel@vger.kernel.org
Am 12.11.2012 16:11, schrieb Sage Weil:
> On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> i was checking what happens if i reboot a ceph node.
>>
>> Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is
>> possible.
>
> If you are using the current master, the new 'min_size' may be biting you;
> ceph osd dump | grep ^pool and see if you see min_size for your pools.
> You can change that back to the norma behavior with
No i don't see any min size:
# ceph osd dump | grep ^pool
pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
1344 pgp_num 1344 last_change 1 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num
1344 pgp_num 1344 last_change 1 owner 0
pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 1344
pgp_num 1344 last_change 1 owner 0
pool 3 'kvmpool1' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
3000 pgp_num 3000 last_change 958 owner 0
> ceph osd pool set <poolname> min_size 1
Yes this helps! But min_size is still not shown in ceph osd dump. Also
when i reboot a node it takes up to 10s-20s until all osds from this
node are set to failed and the I/O starts again. Should i issue an ceph
osd out command before?
But i had already this set for all my rules in my crushmap
min_size 1
max_size 2
in my crushmap for each rule.
Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: ceph cluster hangs when rebooting one node
2012-11-12 15:24 ` Stefan Priebe - Profihost AG
@ 2012-11-14 9:50 ` Aleksey Samarin
2012-11-14 15:06 ` Sage Weil
0 siblings, 1 reply; 5+ messages in thread
From: Aleksey Samarin @ 2012-11-14 9:50 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Sage Weil, ceph-devel@vger.kernel.org
Hello!
I have the same problem. After switching off the second node, the
cluster hangs, there is some solution?
All the best, Alex!
2012/11/12 Stefan Priebe - Profihost AG <s.priebe@profihost.ag>:
> Am 12.11.2012 16:11, schrieb Sage Weil:
>
>> On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote:
>>>
>>> Hello list,
>>>
>>> i was checking what happens if i reboot a ceph node.
>>>
>>> Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is
>>> possible.
>>
>>
>> If you are using the current master, the new 'min_size' may be biting you;
>> ceph osd dump | grep ^pool and see if you see min_size for your pools.
>> You can change that back to the norma behavior with
>
>
> No i don't see any min size:
>
> # ceph osd dump | grep ^pool
> pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 1344
> pgp_num 1344 last_change 1 owner 0 crash_replay_interval 45
> pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num
> 1344 pgp_num 1344 last_change 1 owner 0
> pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 1344
> pgp_num 1344 last_change 1 owner 0
> pool 3 'kvmpool1' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
> 3000 pgp_num 3000 last_change 958 owner 0
>
>
>> ceph osd pool set <poolname> min_size 1
>
> Yes this helps! But min_size is still not shown in ceph osd dump. Also when
> i reboot a node it takes up to 10s-20s until all osds from this node are set
> to failed and the I/O starts again. Should i issue an ceph osd out command
> before?
>
> But i had already this set for all my rules in my crushmap
> min_size 1
> max_size 2
>
> in my crushmap for each rule.
>
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: ceph cluster hangs when rebooting one node
2012-11-14 9:50 ` Aleksey Samarin
@ 2012-11-14 15:06 ` Sage Weil
0 siblings, 0 replies; 5+ messages in thread
From: Sage Weil @ 2012-11-14 15:06 UTC (permalink / raw)
To: Aleksey Samarin; +Cc: Stefan Priebe - Profihost AG, ceph-devel@vger.kernel.org
On Wed, 14 Nov 2012, Aleksey Samarin wrote:
> Hello!
>
> I have the same problem. After switching off the second node, the
> cluster hangs, there is some solution?
>
> All the best, Alex!
I suspect this is min_size; the latest master has a few changes and also
will print it out so you can tell what is going on.
min_size is the minimum number of replicas before the OSDs will go active
(handle reads/writes). Setting it to 1 gets you old behavior, while
increasing it protects you from cases where writes to a single replica
that then fails will force the admin to make a difficult decision about
losing data.
You can adjust with
ceph osd pool set <pool name> min_size <value>
sage
>
> 2012/11/12 Stefan Priebe - Profihost AG <s.priebe@profihost.ag>:
> > Am 12.11.2012 16:11, schrieb Sage Weil:
> >
> >> On Mon, 12 Nov 2012, Stefan Priebe - Profihost AG wrote:
> >>>
> >>> Hello list,
> >>>
> >>> i was checking what happens if i reboot a ceph node.
> >>>
> >>> Sadly if i reboot one node, the whole ceph cluster hangs and no I/O is
> >>> possible.
> >>
> >>
> >> If you are using the current master, the new 'min_size' may be biting you;
> >> ceph osd dump | grep ^pool and see if you see min_size for your pools.
> >> You can change that back to the norma behavior with
> >
> >
> > No i don't see any min size:
> >
> > # ceph osd dump | grep ^pool
> > pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 1344
> > pgp_num 1344 last_change 1 owner 0 crash_replay_interval 45
> > pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num
> > 1344 pgp_num 1344 last_change 1 owner 0
> > pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 1344
> > pgp_num 1344 last_change 1 owner 0
> > pool 3 'kvmpool1' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num
> > 3000 pgp_num 3000 last_change 958 owner 0
> >
> >
> >> ceph osd pool set <poolname> min_size 1
> >
> > Yes this helps! But min_size is still not shown in ceph osd dump. Also when
> > i reboot a node it takes up to 10s-20s until all osds from this node are set
> > to failed and the I/O starts again. Should i issue an ceph osd out command
> > before?
> >
> > But i had already this set for all my rules in my crushmap
> > min_size 1
> > max_size 2
> >
> > in my crushmap for each rule.
> >
> >
> > Stefan
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-11-14 15:06 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-11-12 15:04 ceph cluster hangs when rebooting one node Stefan Priebe - Profihost AG
2012-11-12 15:11 ` Sage Weil
2012-11-12 15:24 ` Stefan Priebe - Profihost AG
2012-11-14 9:50 ` Aleksey Samarin
2012-11-14 15:06 ` Sage Weil
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.