* OSD's slow down to a crawl
@ 2012-12-20 15:16 Matthew Anderson
2012-12-20 15:28 ` Mark Nelson
0 siblings, 1 reply; 13+ messages in thread
From: Matthew Anderson @ 2012-12-20 15:16 UTC (permalink / raw)
To: ceph-devel@vger.kernel.org
Hi All,
I've run into an issue where OSD's slow right down to the point that they no longer appear to be processing write IO and everything grinds to a halt. Once they've stopped performing IO they can be brought back to life by restarting them where they continue on for a while before they slow down again and eventually everything stops again. I'm at a total loss to explain the issue because everything appears to be working correctly yet something is clearly very wrong. Running 'ceph -w' reports everything as clean+active and there are never any slow requests logged.
The problem appears to be more pronounced on the newer 3.7 kernel but happened on previous versions in the same fashion. Weird things like OSD's never completing recovery until they are restarted was also happening occasionally. The cluster would be stuck on a certain percentage degraded and restarting the OSD 3-4 times would eventually see it heal itself with the degraded percentage dropping a little each time. I'm testing with both the kernel CephFS client and QEMU 1.3.0 accessing an RBD, both slow down to a stop in the same manner.
Kernel 3.7.0 mainline (also occurs on 3.7.1 and 3.6.8)
Ceph version 0.55.1-300-g08c6424 (08c64249eb8cd7922de5c398a9426538918db77c) (also occurs on 0.55 and 0.55.1)
Total of 3 hosts. Two of them with 4 OSD's on an SSD each. The other with 24x 2TB SATA drives. Each host is running a monitor. I have setup two pools in the CRUSHMAP, one for SATA and one for SSD. Both pools have the same problem.
OSD config:
[osd.24]
host = KVM04
osd data = /srv/ceph/osd/$name
btrfs devs = /dev/sda2
btrfs options = rw,noatime,ssd
osd journal = /dev/sda1
The log below is when the OSD has stopped processing IO and was then shutdown cleanly. The QEMU guest was partially responsive but the benchmark had frozen waiting for IO to complete.
If I can do anymore debugging to help diagnose the problem please let me know.
Thanks
-Matt
-------------------------------------
2012-12-20 22:18:06.809863 7fb9e4ff9700 20 journal write_thread_entry aio throttle: aio num 0 bytes 0 ... exp 0 min_new 1 ... pending 5748
2012-12-20 22:18:06.809871 7fb9e4ff9700 10 journal room 960479231 max_size 960495616 pos 543215616 header.start 543203328 top 4096
2012-12-20 22:18:06.809877 7fb9e4ff9700 10 journal check_for_full at 543215616 : 8192 < 960479231
2012-12-20 22:18:06.809880 7fb9e4ff9700 15 journal prepare_single_write 1 will write 543215616 : seq 2087065 len 5748 -> 8192 (head 40 pre_pad 0 ebl 5748 post_pad 2364 tail 40) (ebl alignment -1)
2012-12-20 22:18:06.809891 7fb9d57fa700 10 journal op_apply_start 2087065 open_ops 0 -> 1
2012-12-20 22:18:06.809897 7fb9d57fa700 5 filestore(/srv/ceph/osd/osd.24) _do_op 0x7fb9b4376570 seq 2087065 osr(6.1fe3 0x21a4f50)/0x21a4f50 start
2012-12-20 22:18:06.809901 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _do_transaction on 0x7fb9b4f5db40
2012-12-20 22:18:06.809915 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) write meta/a4c1c2a3/pglog_6.1fe3/0//-1 147735~147
2012-12-20 22:18:06.809947 7fb9e4ff9700 20 journal prepare_multi_write queue_pos now 543223808
2012-12-20 22:18:06.809953 7fb9e4ff9700 15 journal do_aio_write writing 543215616~8192
2012-12-20 22:18:06.809975 7fb9e4ff9700 20 journal write_aio_bl 543215616~8192 seq 2087065
2012-12-20 22:18:06.809981 7fb9e4ff9700 20 journal write_aio_bl .. 543215616~8192 in 1
2012-12-20 22:18:06.810051 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) write meta/a4c1c2a3/pglog_6.1fe3/0//-1 147735~147 = 147
2012-12-20 22:18:06.810065 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'ondisklog' len 34
2012-12-20 22:18:06.810068 7fb9e4ff9700 5 journal put_throttle finished 1 ops and 5748 bytes, now 0 ops and 0 bytes
2012-12-20 22:18:06.810077 7fb9e4ff9700 20 journal write_thread_entry going to sleep
2012-12-20 22:18:06.810103 7fb9d7fff700 20 journal write_finish_thread_entry waiting for aio(s)
2012-12-20 22:18:06.810149 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'ondisklog' len 34 = 34
2012-12-20 22:18:06.810159 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'info' len 5
2012-12-20 22:18:06.810191 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'info' len 5 = 5
2012-12-20 22:18:06.810198 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) truncate meta/8dfefeef/pginfo_6.1fe3/0//-1 size 0
2012-12-20 22:18:06.810307 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) truncate meta/8dfefeef/pginfo_6.1fe3/0//-1 size 0 = 0
2012-12-20 22:18:06.810317 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) write meta/8dfefeef/pginfo_6.1fe3/0//-1 0~504
2012-12-20 22:18:06.810386 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) write meta/8dfefeef/pginfo_6.1fe3/0//-1 0~504 = 504
2012-12-20 22:18:06.810394 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _do_transaction on 0x7fb9b4f5dab8
2012-12-20 22:18:06.810399 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) write 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 3522560~4096
2012-12-20 22:18:06.810451 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) write 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 3522560~4096 = 4096
2012-12-20 22:18:06.810497 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6
2012-12-20 22:18:06.810576 7fb9d7fff700 10 journal write_finish_thread_entry aio 543215616~8192 done
2012-12-20 22:18:06.810588 7fb9d7fff700 20 journal check_aio_completion
2012-12-20 22:18:06.810590 7fb9d7fff700 20 journal check_aio_completion completed seq 2087065 543215616~8192
2012-12-20 22:18:06.810594 7fb9d7fff700 20 journal check_aio_completion queueing finishers through seq 2087065
2012-12-20 22:18:06.810596 7fb9d7fff700 10 journal queue_completions_thru seq 2087065 queueing seq 2087065 0x7fb9b4f70ac0 lat 0.000761
2012-12-20 22:18:06.810631 7fb9d7fff700 20 journal write_finish_thread_entry sleeping
2012-12-20 22:18:06.810695 7fb9d5ffb700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 luod=3473'23688 lcod 3473'23688 mlcod 3473'23687 active+clean] op_commit repgather(0x7fb9b47d1050 applying 3473'23689 rep_tid=72 wfack=24,31 wfdisk=24,31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
2012-12-20 22:18:06.810720 7fb9d5ffb700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] eval_repop repgather(0x7fb9b47d1050 applying 3473'23689 rep_tid=72 wfack=31 wfdisk=31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) wants=ad
2012-12-20 22:18:06.810992 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 = 230
2012-12-20 22:18:06.811053 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6
2012-12-20 22:18:06.811083 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 = 31
2012-12-20 22:18:06.811093 7fb9d57fa700 10 journal op_apply_finish 2087065 open_ops 1 -> 0, max_applied_seq 2087064 -> 2087065
2012-12-20 22:18:06.811096 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _do_op 0x7fb9b4376570 seq 2087065 r = 0, finisher 0x7fb9b5022bb0 0x7fb9b483ce10
2012-12-20 22:18:06.811100 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _finish_op 0x7fb9b4376570 seq 2087065 osr(6.1fe3 0x21a4f50)/0x21a4f50
2012-12-20 22:18:06.811136 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] op_applied repgather(0x7fb9b47d1050 applying 3473'23689 rep_tid=72 wfack=31 wfdisk=31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
2012-12-20 22:18:06.811205 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] op_applied mode was rmw(wr=1)
2012-12-20 22:18:06.811218 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] op_applied mode now idle(wr=0 WAKE) (finish_write)
2012-12-20 22:18:06.811229 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] put_object_context 0x7fb9b5248020 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 1 -> 0
2012-12-20 22:18:06.811241 7fb9c37fe700 15 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] requeue_ops
2012-12-20 22:18:06.811251 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] put_snapset_context rb.0.122c.6b8b4567.000000000357 1 -> 0
2012-12-20 22:18:06.811267 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] eval_repop repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack=31 wfdisk=31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) wants=ad
2012-12-20 22:18:06.813738 7fb9c0ff9700 20 osd.24 3473 _dispatch 0x7fb9600008f0 osd_sub_op_reply(client.7636.0:100075 6.1fe3 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 [] ondisk, result = 0) v1
2012-12-20 22:18:06.813850 7fb9c0ff9700 15 osd.24 3473 require_same_or_newer_map 3473 (i am 3473) 0x7fb9600008f0
2012-12-20 22:18:06.813856 7fb9c0ff9700 20 osd.24 3473 _share_map_incoming osd.31 172.16.0.12:6820/2871 3473
2012-12-20 22:18:06.813865 7fb9c0ff9700 15 osd.24 3473 enqueue_op 0x7fb968055650 osd_sub_op_reply(client.7636.0:100075 6.1fe3 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 [] ondisk,
result= 0) v1
2012-12-20 22:18:06.813934 7fb9bd7f2700 10 osd.24 3473 dequeue_op 0x7fb968055650 osd_sub_op_reply(client.7636.0:100075 6.1fe3 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 [] ondisk, result = 0) v1 pg pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean]
2012-12-20 22:18:06.814005 7fb9bd7f2700 7 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] repop_ack rep_tid 72 op osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4 result 0 ack_type 4 from osd.31
2012-12-20 22:18:06.814037 7fb9bd7f2700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] eval_repop repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) wants=ad
2012-12-20 22:18:06.814057 7fb9bd7f2700 15 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] log_op_stats osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4 inb 4239 outb 0 rlat 0.000000 lat 0.005594
2012-12-20 22:18:06.814075 7fb9bd7f2700 15 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] update_stats 3465'46631
2012-12-20 22:18:06.814088 7fb9bd7f2700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] sending commit on repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) 0x7fb9b4605310
2012-12-20 22:18:06.814122 7fb9bd7f2700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23688 active+clean] removing repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
2012-12-20 22:18:06.814147 7fb9bd7f2700 20 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23688 active+clean] q front is repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
2012-12-20 22:18:06.814204 7fb9bd7f2700 10 osd.24 3473 dequeue_op 0x7fb968055650 finish
2012-12-20 22:18:07.030914 7fb9be7f4700 10 monclient: tick
2012-12-20 22:18:07.030929 7fb9be7f4700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2012-12-20 22:17:37.030927)
2012-12-20 22:18:07.030952 7fb9be7f4700 10 monclient: renew subs? (now: 2012-12-20 22:18:07.030952; renew after: 2012-12-20 22:18:33.322998) -- no
2012-12-20 22:18:07.172960 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:07.172979 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:07.173647 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:07.173675 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:07.173785 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:07.173875 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:07.479259 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:07.479311 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
2012-12-20 22:18:07.479326 7fb9e6ffd700 7 osd.24 3473 do_mon_report
2012-12-20 22:18:07.479329 7fb9e6ffd700 10 osd.24 3473 send_alive up_thru currently 3465 want 3465
2012-12-20 22:18:07.479331 7fb9e6ffd700 20 osd.24 3473 send_pg_stats
2012-12-20 22:18:07.479334 7fb9e6ffd700 10 osd.24 3473 send_pg_stats - 2 pgs updated
2012-12-20 22:18:07.479343 7fb9e6ffd700 10 monclient: _send_mon_message to mon.KVM03 at 172.16.0.12:6789/0
2012-12-20 22:18:07.774794 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:07.809251 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 5.000117
2012-12-20 22:18:07.809274 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
2012-12-20 22:18:07.809278 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
2012-12-20 22:18:07.809279 7fb9e57fa700 10 journal commit_start committing 2087065, still blocked
2012-12-20 22:18:07.809282 7fb9e57fa700 10 journal commit_start
2012-12-20 22:18:07.809291 7fb9e57fa700 15 filestore(/srv/ceph/osd/osd.24) sync_entry committing 2087065 sync_epoch 19
2012-12-20 22:18:07.809339 7fb9e57fa700 10 filestore(/srv/ceph/osd/osd.24) taking async snap 'snap_2087065'
2012-12-20 22:18:07.817045 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) async snap create 'snap_2087065' transid 96549
2012-12-20 22:18:07.817060 7fb9e57fa700 10 journal commit_started committing 2087065, unblocking
2012-12-20 22:18:07.817069 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) waiting for transid 96549 to complete
2012-12-20 22:18:07.836752 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) done waiting for transid 96549 to complete
2012-12-20 22:18:07.836767 7fb9e57fa700 10 filestore(/srv/ceph/osd/osd.24) sync_entry commit took 0.027482, interval was 5.027633
2012-12-20 22:18:07.836775 7fb9e57fa700 10 journal commit_finish thru 2087065
2012-12-20 22:18:07.836778 7fb9e57fa700 5 journal committed_thru 2087065 (last_committed_seq 2087063)
2012-12-20 22:18:07.836781 7fb9e57fa700 10 journal header: block_size 4096 alignment 4096 max_size 960495616
2012-12-20 22:18:07.836783 7fb9e57fa700 10 journal header: start 543223808
2012-12-20 22:18:07.836784 7fb9e57fa700 10 journal write_pos 543223808
2012-12-20 22:18:07.836788 7fb9e57fa700 10 journal committed_thru done
2012-12-20 22:18:07.836793 7fb9e57fa700 10 filestore(/srv/ceph/osd/osd.24) removing snap 'snap_2087061'
2012-12-20 22:18:07.836958 7fb9e57fa700 15 filestore(/srv/ceph/osd/osd.24) sync_entry committed to op_seq 2087065
2012-12-20 22:18:07.836972 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
2012-12-20 22:18:07.855890 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:08.221094 7fb9c1ffb700 20 osd.24 3473 _dispatch 0x7fb990002de0 pg_stats_ack(2 pgs tid 18) v1
2012-12-20 22:18:08.221105 7fb9c1ffb700 10 osd.24 3473 handle_pg_stats_ack
2012-12-20 22:18:08.479450 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:08.479499 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
2012-12-20 22:18:09.479581 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:09.479633 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
2012-12-20 22:18:10.479743 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:10.479808 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
2012-12-20 22:18:11.273304 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:11.273324 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:11.274018 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:11.274081 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:11.274101 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:11.274249 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:11.275112 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:11.356141 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:11.479927 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:11.479976 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
2012-12-20 22:18:12.182095 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:12.373677 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:12.373694 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:12.374496 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:12.374523 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:12.374541 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:12.374564 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:12.480076 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:12.480125 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
2012-12-20 22:18:12.480140 7fb9e6ffd700 7 osd.24 3473 do_mon_report
2012-12-20 22:18:12.480142 7fb9e6ffd700 10 osd.24 3473 send_alive up_thru currently 3465 want 3465
2012-12-20 22:18:12.480146 7fb9e6ffd700 20 osd.24 3473 send_pg_stats
2012-12-20 22:18:12.701202 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:12.837067 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 5.000094
2012-12-20 22:18:12.837089 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
2012-12-20 22:18:12.837093 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
2012-12-20 22:18:12.837095 7fb9e57fa700 10 journal commit_start nothing to do
2012-12-20 22:18:12.837100 7fb9e57fa700 10 journal commit_start
2012-12-20 22:18:12.837117 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
2012-12-20 22:18:13.480239 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:13.480289 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
2012-12-20 22:18:14.073917 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:14.073935 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:14.074840 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:14.074868 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:14.074886 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:14.074903 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:14.480386 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:14.480443 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
2012-12-20 22:18:15.480571 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:15.480632 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
2012-12-20 22:18:16.282625 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:16.480714 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:16.480763 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
2012-12-20 22:18:16.801391 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:17.031056 7fb9be7f4700 10 monclient: tick
2012-12-20 22:18:17.031082 7fb9be7f4700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2012-12-20 22:17:47.031080)
2012-12-20 22:18:17.031124 7fb9be7f4700 10 monclient: renew subs? (now: 2012-12-20 22:18:17.031124; renew after: 2012-12-20 22:18:33.322998) -- no
2012-12-20 22:18:17.175266 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:17.256256 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:17.480898 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:17.480952 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
2012-12-20 22:18:17.480969 7fb9e6ffd700 7 osd.24 3473 do_mon_report
2012-12-20 22:18:17.480972 7fb9e6ffd700 10 osd.24 3473 send_alive up_thru currently 3465 want 3465
2012-12-20 22:18:17.480975 7fb9e6ffd700 20 osd.24 3473 send_pg_stats
2012-12-20 22:18:17.837232 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 5.000116
2012-12-20 22:18:17.837254 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
2012-12-20 22:18:17.837258 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
2012-12-20 22:18:17.837260 7fb9e57fa700 10 journal commit_start nothing to do
2012-12-20 22:18:17.837262 7fb9e57fa700 10 journal commit_start
2012-12-20 22:18:17.837271 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
2012-12-20 22:18:18.481073 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:18.481133 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
2012-12-20 22:18:19.481247 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:19.481308 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
2012-12-20 22:18:19.974219 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:19.974238 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
2012-12-20 22:18:19.974899 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
2012-12-20 22:18:19.975070 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
2012-12-20 22:18:19.975116 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
2012-12-20 22:18:19.975207 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
2012-12-20 22:18:20.481425 7fb9e6ffd700 5 osd.24 3473 tick
2012-12-20 22:18:20.481487 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
2012-12-20 22:18:21.016361 7fb9e77fe700 -1 osd.24 3473 *** Got signal Terminated ***
2012-12-20 22:18:21.016458 7fb9e77fe700 -1 osd.24 3473 pausing thread pools
2012-12-20 22:18:21.016462 7fb9e77fe700 -1 osd.24 3473 flushing io
2012-12-20 22:18:21.016464 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync_and_flush
2012-12-20 22:18:21.016467 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) _flush_op_queue draining op tp
2012-12-20 22:18:21.016471 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) _flush_op_queue waiting for apply finisher
2012-12-20 22:18:21.016481 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) start_sync
2012-12-20 22:18:21.016483 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync waiting
2012-12-20 22:18:21.016523 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 3.179252
2012-12-20 22:18:21.016540 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
2012-12-20 22:18:21.016543 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
2012-12-20 22:18:21.016545 7fb9e57fa700 10 journal commit_start nothing to do
2012-12-20 22:18:21.016548 7fb9e57fa700 10 journal commit_start
2012-12-20 22:18:21.016563 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
2012-12-20 22:18:21.016563 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync done
2012-12-20 22:18:21.016570 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync_and_flush done
2012-12-20 22:18:21.016572 7fb9e77fe700 -1 osd.24 3473 removing pid file
2012-12-20 22:18:21.016665 7fb9e77fe700 -1 osd.24 3473 exit
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: OSD's slow down to a crawl
2012-12-20 15:16 OSD's slow down to a crawl Matthew Anderson
@ 2012-12-20 15:28 ` Mark Nelson
2012-12-20 15:51 ` Matthew Anderson
0 siblings, 1 reply; 13+ messages in thread
From: Mark Nelson @ 2012-12-20 15:28 UTC (permalink / raw)
To: Matthew Anderson; +Cc: ceph-devel@vger.kernel.org
On 12/20/2012 09:16 AM, Matthew Anderson wrote:
> Hi All,
>
> I've run into an issue where OSD's slow right down to the point that they no longer appear to be processing write IO and everything grinds to a halt. Once they've stopped performing IO they can be brought back to life by restarting them where they continue on for a while before they slow down again and eventually everything stops again. I'm at a total loss to explain the issue because everything appears to be working correctly yet something is clearly very wrong. Running 'ceph -w' reports everything as clean+active and there are never any slow requests logged.
>
> The problem appears to be more pronounced on the newer 3.7 kernel but happened on previous versions in the same fashion. Weird things like OSD's never completing recovery until they are restarted was also happening occasionally. The cluster would be stuck on a certain percentage degraded and restarting the OSD 3-4 times would eventually see it heal itself with the degraded percentage dropping a little each time. I'm testing with both the kernel CephFS client and QEMU 1.3.0 accessing an RBD, both slow down to a stop in the same manner.
>
> Kernel 3.7.0 mainline (also occurs on 3.7.1 and 3.6.8)
> Ceph version 0.55.1-300-g08c6424 (08c64249eb8cd7922de5c398a9426538918db77c) (also occurs on 0.55 and 0.55.1)
> Total of 3 hosts. Two of them with 4 OSD's on an SSD each. The other with 24x 2TB SATA drives. Each host is running a monitor. I have setup two pools in the CRUSHMAP, one for SATA and one for SSD. Both pools have the same problem.
>
> OSD config:
> [osd.24]
> host = KVM04
> osd data = /srv/ceph/osd/$name
> btrfs devs = /dev/sda2
> btrfs options = rw,noatime,ssd
> osd journal = /dev/sda1
>
> The log below is when the OSD has stopped processing IO and was then shutdown cleanly. The QEMU guest was partially responsive but the benchmark had frozen waiting for IO to complete.
>
> If I can do anymore debugging to help diagnose the problem please let me know.
Hi Matt,
Yeah, your log is looking pretty sparse! I imagine Sam will have a lot
of questions for you. Out of curiosity have you looked at any
io/cpu/memory stats when things slow down? sysprof may come in handy if
it looks like it's CPU related. Also, check out the OSD admin socket.
Sebastien Han made a nice little blog entry describing it:
http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
You can get a lot of information about the current state of things.
Mark
>
> Thanks
> -Matt
> -------------------------------------
>
> 2012-12-20 22:18:06.809863 7fb9e4ff9700 20 journal write_thread_entry aio throttle: aio num 0 bytes 0 ... exp 0 min_new 1 ... pending 5748
> 2012-12-20 22:18:06.809871 7fb9e4ff9700 10 journal room 960479231 max_size 960495616 pos 543215616 header.start 543203328 top 4096
> 2012-12-20 22:18:06.809877 7fb9e4ff9700 10 journal check_for_full at 543215616 : 8192 < 960479231
> 2012-12-20 22:18:06.809880 7fb9e4ff9700 15 journal prepare_single_write 1 will write 543215616 : seq 2087065 len 5748 -> 8192 (head 40 pre_pad 0 ebl 5748 post_pad 2364 tail 40) (ebl alignment -1)
> 2012-12-20 22:18:06.809891 7fb9d57fa700 10 journal op_apply_start 2087065 open_ops 0 -> 1
> 2012-12-20 22:18:06.809897 7fb9d57fa700 5 filestore(/srv/ceph/osd/osd.24) _do_op 0x7fb9b4376570 seq 2087065 osr(6.1fe3 0x21a4f50)/0x21a4f50 start
> 2012-12-20 22:18:06.809901 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _do_transaction on 0x7fb9b4f5db40
> 2012-12-20 22:18:06.809915 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) write meta/a4c1c2a3/pglog_6.1fe3/0//-1 147735~147
> 2012-12-20 22:18:06.809947 7fb9e4ff9700 20 journal prepare_multi_write queue_pos now 543223808
> 2012-12-20 22:18:06.809953 7fb9e4ff9700 15 journal do_aio_write writing 543215616~8192
> 2012-12-20 22:18:06.809975 7fb9e4ff9700 20 journal write_aio_bl 543215616~8192 seq 2087065
> 2012-12-20 22:18:06.809981 7fb9e4ff9700 20 journal write_aio_bl .. 543215616~8192 in 1
> 2012-12-20 22:18:06.810051 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) write meta/a4c1c2a3/pglog_6.1fe3/0//-1 147735~147 = 147
> 2012-12-20 22:18:06.810065 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'ondisklog' len 34
> 2012-12-20 22:18:06.810068 7fb9e4ff9700 5 journal put_throttle finished 1 ops and 5748 bytes, now 0 ops and 0 bytes
> 2012-12-20 22:18:06.810077 7fb9e4ff9700 20 journal write_thread_entry going to sleep
> 2012-12-20 22:18:06.810103 7fb9d7fff700 20 journal write_finish_thread_entry waiting for aio(s)
> 2012-12-20 22:18:06.810149 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'ondisklog' len 34 = 34
> 2012-12-20 22:18:06.810159 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'info' len 5
> 2012-12-20 22:18:06.810191 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) collection_setattr /srv/ceph/osd/osd.24/current/6.1fe3_head 'info' len 5 = 5
> 2012-12-20 22:18:06.810198 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) truncate meta/8dfefeef/pginfo_6.1fe3/0//-1 size 0
> 2012-12-20 22:18:06.810307 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) truncate meta/8dfefeef/pginfo_6.1fe3/0//-1 size 0 = 0
> 2012-12-20 22:18:06.810317 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) write meta/8dfefeef/pginfo_6.1fe3/0//-1 0~504
> 2012-12-20 22:18:06.810386 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) write meta/8dfefeef/pginfo_6.1fe3/0//-1 0~504 = 504
> 2012-12-20 22:18:06.810394 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _do_transaction on 0x7fb9b4f5dab8
> 2012-12-20 22:18:06.810399 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) write 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 3522560~4096
> 2012-12-20 22:18:06.810451 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) write 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 3522560~4096 = 4096
> 2012-12-20 22:18:06.810497 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6
> 2012-12-20 22:18:06.810576 7fb9d7fff700 10 journal write_finish_thread_entry aio 543215616~8192 done
> 2012-12-20 22:18:06.810588 7fb9d7fff700 20 journal check_aio_completion
> 2012-12-20 22:18:06.810590 7fb9d7fff700 20 journal check_aio_completion completed seq 2087065 543215616~8192
> 2012-12-20 22:18:06.810594 7fb9d7fff700 20 journal check_aio_completion queueing finishers through seq 2087065
> 2012-12-20 22:18:06.810596 7fb9d7fff700 10 journal queue_completions_thru seq 2087065 queueing seq 2087065 0x7fb9b4f70ac0 lat 0.000761
> 2012-12-20 22:18:06.810631 7fb9d7fff700 20 journal write_finish_thread_entry sleeping
> 2012-12-20 22:18:06.810695 7fb9d5ffb700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 luod=3473'23688 lcod 3473'23688 mlcod 3473'23687 active+clean] op_commit repgather(0x7fb9b47d1050 applying 3473'23689 rep_tid=72 wfack=24,31 wfdisk=24,31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
> 2012-12-20 22:18:06.810720 7fb9d5ffb700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] eval_repop repgather(0x7fb9b47d1050 applying 3473'23689 rep_tid=72 wfack=31 wfdisk=31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) wants=ad
> 2012-12-20 22:18:06.810992 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 = 230
> 2012-12-20 22:18:06.811053 7fb9d57fa700 15 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6
> 2012-12-20 22:18:06.811083 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) setattrs 6.1fe3_head/42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 = 31
> 2012-12-20 22:18:06.811093 7fb9d57fa700 10 journal op_apply_finish 2087065 open_ops 1 -> 0, max_applied_seq 2087064 -> 2087065
> 2012-12-20 22:18:06.811096 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _do_op 0x7fb9b4376570 seq 2087065 r = 0, finisher 0x7fb9b5022bb0 0x7fb9b483ce10
> 2012-12-20 22:18:06.811100 7fb9d57fa700 10 filestore(/srv/ceph/osd/osd.24) _finish_op 0x7fb9b4376570 seq 2087065 osr(6.1fe3 0x21a4f50)/0x21a4f50
> 2012-12-20 22:18:06.811136 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] op_applied repgather(0x7fb9b47d1050 applying 3473'23689 rep_tid=72 wfack=31 wfdisk=31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
> 2012-12-20 22:18:06.811205 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] op_applied mode was rmw(wr=1)
> 2012-12-20 22:18:06.811218 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] op_applied mode now idle(wr=0 WAKE) (finish_write)
> 2012-12-20 22:18:06.811229 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] put_object_context 0x7fb9b5248020 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 1 -> 0
> 2012-12-20 22:18:06.811241 7fb9c37fe700 15 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] requeue_ops
> 2012-12-20 22:18:06.811251 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] put_snapset_context rb.0.122c.6b8b4567.000000000357 1 -> 0
> 2012-12-20 22:18:06.811267 7fb9c37fe700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] eval_repop repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack=31 wfdisk=31 op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) wants=ad
> 2012-12-20 22:18:06.813738 7fb9c0ff9700 20 osd.24 3473 _dispatch 0x7fb9600008f0 osd_sub_op_reply(client.7636.0:100075 6.1fe3 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 [] ondisk, result = 0) v1
> 2012-12-20 22:18:06.813850 7fb9c0ff9700 15 osd.24 3473 require_same_or_newer_map 3473 (i am 3473) 0x7fb9600008f0
> 2012-12-20 22:18:06.813856 7fb9c0ff9700 20 osd.24 3473 _share_map_incoming osd.31 172.16.0.12:6820/2871 3473
> 2012-12-20 22:18:06.813865 7fb9c0ff9700 15 osd.24 3473 enqueue_op 0x7fb968055650 osd_sub_op_reply(client.7636.0:100075 6.1fe3 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 [] ondisk,
> result= 0) v1
> 2012-12-20 22:18:06.813934 7fb9bd7f2700 10 osd.24 3473 dequeue_op 0x7fb968055650 osd_sub_op_reply(client.7636.0:100075 6.1fe3 42463fe3/rb.0.122c.6b8b4567.000000000357/head//6 [] ondisk, result = 0) v1 pg pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean]
> 2012-12-20 22:18:06.814005 7fb9bd7f2700 7 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] repop_ack rep_tid 72 op osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4 result 0 ack_type 4 from osd.31
> 2012-12-20 22:18:06.814037 7fb9bd7f2700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] eval_repop repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) wants=ad
> 2012-12-20 22:18:06.814057 7fb9bd7f2700 15 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] log_op_stats osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4 inb 4239 outb 0 rlat 0.000000 lat 0.005594
> 2012-12-20 22:18:06.814075 7fb9bd7f2700 15 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] update_stats 3465'46631
> 2012-12-20 22:18:06.814088 7fb9bd7f2700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23687 active+clean] sending commit on repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4) 0x7fb9b4605310
> 2012-12-20 22:18:06.814122 7fb9bd7f2700 10 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23688 active+clean] removing repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
> 2012-12-20 22:18:06.814147 7fb9bd7f2700 20 osd.24 pg_epoch: 3473 pg[6.1fe3( v 3473'23689 (3247'22688,3473'23689] local-les=3472 n=12 ec=656 les/c 3472/3473 3465/3465/3465) [24,31] r=0 lpr=3465 mlcod 3473'23688 active+clean] q front is repgather(0x7fb9b47d1050 applied 3473'23689 rep_tid=72 wfack= wfdisk= op=osd_op(client.7636.0:100075 rb.0.122c.6b8b4567.000000000357 [write 3522560~4096] 6.42463fe3) v4)
> 2012-12-20 22:18:06.814204 7fb9bd7f2700 10 osd.24 3473 dequeue_op 0x7fb968055650 finish
> 2012-12-20 22:18:07.030914 7fb9be7f4700 10 monclient: tick
> 2012-12-20 22:18:07.030929 7fb9be7f4700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2012-12-20 22:17:37.030927)
> 2012-12-20 22:18:07.030952 7fb9be7f4700 10 monclient: renew subs? (now: 2012-12-20 22:18:07.030952; renew after: 2012-12-20 22:18:33.322998) -- no
> 2012-12-20 22:18:07.172960 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:07.172979 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:07.173647 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:07.173675 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:07.173785 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:07.173875 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:07.479259 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:07.479311 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
> 2012-12-20 22:18:07.479326 7fb9e6ffd700 7 osd.24 3473 do_mon_report
> 2012-12-20 22:18:07.479329 7fb9e6ffd700 10 osd.24 3473 send_alive up_thru currently 3465 want 3465
> 2012-12-20 22:18:07.479331 7fb9e6ffd700 20 osd.24 3473 send_pg_stats
> 2012-12-20 22:18:07.479334 7fb9e6ffd700 10 osd.24 3473 send_pg_stats - 2 pgs updated
> 2012-12-20 22:18:07.479343 7fb9e6ffd700 10 monclient: _send_mon_message to mon.KVM03 at 172.16.0.12:6789/0
> 2012-12-20 22:18:07.774794 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:07.809251 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 5.000117
> 2012-12-20 22:18:07.809274 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
> 2012-12-20 22:18:07.809278 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
> 2012-12-20 22:18:07.809279 7fb9e57fa700 10 journal commit_start committing 2087065, still blocked
> 2012-12-20 22:18:07.809282 7fb9e57fa700 10 journal commit_start
> 2012-12-20 22:18:07.809291 7fb9e57fa700 15 filestore(/srv/ceph/osd/osd.24) sync_entry committing 2087065 sync_epoch 19
> 2012-12-20 22:18:07.809339 7fb9e57fa700 10 filestore(/srv/ceph/osd/osd.24) taking async snap 'snap_2087065'
> 2012-12-20 22:18:07.817045 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) async snap create 'snap_2087065' transid 96549
> 2012-12-20 22:18:07.817060 7fb9e57fa700 10 journal commit_started committing 2087065, unblocking
> 2012-12-20 22:18:07.817069 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) waiting for transid 96549 to complete
> 2012-12-20 22:18:07.836752 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) done waiting for transid 96549 to complete
> 2012-12-20 22:18:07.836767 7fb9e57fa700 10 filestore(/srv/ceph/osd/osd.24) sync_entry commit took 0.027482, interval was 5.027633
> 2012-12-20 22:18:07.836775 7fb9e57fa700 10 journal commit_finish thru 2087065
> 2012-12-20 22:18:07.836778 7fb9e57fa700 5 journal committed_thru 2087065 (last_committed_seq 2087063)
> 2012-12-20 22:18:07.836781 7fb9e57fa700 10 journal header: block_size 4096 alignment 4096 max_size 960495616
> 2012-12-20 22:18:07.836783 7fb9e57fa700 10 journal header: start 543223808
> 2012-12-20 22:18:07.836784 7fb9e57fa700 10 journal write_pos 543223808
> 2012-12-20 22:18:07.836788 7fb9e57fa700 10 journal committed_thru done
> 2012-12-20 22:18:07.836793 7fb9e57fa700 10 filestore(/srv/ceph/osd/osd.24) removing snap 'snap_2087061'
> 2012-12-20 22:18:07.836958 7fb9e57fa700 15 filestore(/srv/ceph/osd/osd.24) sync_entry committed to op_seq 2087065
> 2012-12-20 22:18:07.836972 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
> 2012-12-20 22:18:07.855890 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:08.221094 7fb9c1ffb700 20 osd.24 3473 _dispatch 0x7fb990002de0 pg_stats_ack(2 pgs tid 18) v1
> 2012-12-20 22:18:08.221105 7fb9c1ffb700 10 osd.24 3473 handle_pg_stats_ack
> 2012-12-20 22:18:08.479450 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:08.479499 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
> 2012-12-20 22:18:09.479581 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:09.479633 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
> 2012-12-20 22:18:10.479743 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:10.479808 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
> 2012-12-20 22:18:11.273304 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:11.273324 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:11.274018 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:11.274081 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:11.274101 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:11.274249 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:11.275112 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:11.356141 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:11.479927 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:11.479976 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.36 >= max 0.5 = no, load too high
> 2012-12-20 22:18:12.182095 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:12.373677 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:12.373694 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:12.374496 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:12.374523 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:12.374541 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:12.374564 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:12.480076 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:12.480125 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
> 2012-12-20 22:18:12.480140 7fb9e6ffd700 7 osd.24 3473 do_mon_report
> 2012-12-20 22:18:12.480142 7fb9e6ffd700 10 osd.24 3473 send_alive up_thru currently 3465 want 3465
> 2012-12-20 22:18:12.480146 7fb9e6ffd700 20 osd.24 3473 send_pg_stats
> 2012-12-20 22:18:12.701202 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:12.837067 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 5.000094
> 2012-12-20 22:18:12.837089 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
> 2012-12-20 22:18:12.837093 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
> 2012-12-20 22:18:12.837095 7fb9e57fa700 10 journal commit_start nothing to do
> 2012-12-20 22:18:12.837100 7fb9e57fa700 10 journal commit_start
> 2012-12-20 22:18:12.837117 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
> 2012-12-20 22:18:13.480239 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:13.480289 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
> 2012-12-20 22:18:14.073917 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:14.073935 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:14.074840 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:14.074868 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:14.074886 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:14.074903 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:14.480386 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:14.480443 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
> 2012-12-20 22:18:15.480571 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:15.480632 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
> 2012-12-20 22:18:16.282625 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:16.480714 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:16.480763 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 6.09 >= max 0.5 = no, load too high
> 2012-12-20 22:18:16.801391 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:17.031056 7fb9be7f4700 10 monclient: tick
> 2012-12-20 22:18:17.031082 7fb9be7f4700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2012-12-20 22:17:47.031080)
> 2012-12-20 22:18:17.031124 7fb9be7f4700 10 monclient: renew subs? (now: 2012-12-20 22:18:17.031124; renew after: 2012-12-20 22:18:33.322998) -- no
> 2012-12-20 22:18:17.175266 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:17.256256 7fb9bf7f6700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:17.480898 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:17.480952 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
> 2012-12-20 22:18:17.480969 7fb9e6ffd700 7 osd.24 3473 do_mon_report
> 2012-12-20 22:18:17.480972 7fb9e6ffd700 10 osd.24 3473 send_alive up_thru currently 3465 want 3465
> 2012-12-20 22:18:17.480975 7fb9e6ffd700 20 osd.24 3473 send_pg_stats
> 2012-12-20 22:18:17.837232 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 5.000116
> 2012-12-20 22:18:17.837254 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
> 2012-12-20 22:18:17.837258 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
> 2012-12-20 22:18:17.837260 7fb9e57fa700 10 journal commit_start nothing to do
> 2012-12-20 22:18:17.837262 7fb9e57fa700 10 journal commit_start
> 2012-12-20 22:18:17.837271 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
> 2012-12-20 22:18:18.481073 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:18.481133 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
> 2012-12-20 22:18:19.481247 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:19.481308 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
> 2012-12-20 22:18:19.974219 7fb9bafed700 20 osd.24 3473 update_osd_stat osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:19.974238 7fb9bafed700 5 osd.24 3473 heartbeat: osd_stat(21111 MB used, 393 GB avail, 431 GB total, peers [28,29,30,31]/[])
> 2012-12-20 22:18:19.974899 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a002f010 already has epoch 3473
> 2012-12-20 22:18:19.975070 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a00334d0 already has epoch 3473
> 2012-12-20 22:18:19.975116 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9a0033fc0 already has epoch 3473
> 2012-12-20 22:18:19.975207 7fb9bfff7700 20 osd.24 3473 _share_map_outgoing 0x7fb9b400d440 already has epoch 3473
> 2012-12-20 22:18:20.481425 7fb9e6ffd700 5 osd.24 3473 tick
> 2012-12-20 22:18:20.481487 7fb9e6ffd700 20 osd.24 3473 scrub_should_schedule loadavg 5.84 >= max 0.5 = no, load too high
> 2012-12-20 22:18:21.016361 7fb9e77fe700 -1 osd.24 3473 *** Got signal Terminated ***
> 2012-12-20 22:18:21.016458 7fb9e77fe700 -1 osd.24 3473 pausing thread pools
> 2012-12-20 22:18:21.016462 7fb9e77fe700 -1 osd.24 3473 flushing io
> 2012-12-20 22:18:21.016464 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync_and_flush
> 2012-12-20 22:18:21.016467 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) _flush_op_queue draining op tp
> 2012-12-20 22:18:21.016471 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) _flush_op_queue waiting for apply finisher
> 2012-12-20 22:18:21.016481 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) start_sync
> 2012-12-20 22:18:21.016483 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync waiting
> 2012-12-20 22:18:21.016523 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry woke after 3.179252
> 2012-12-20 22:18:21.016540 7fb9e57fa700 10 journal commit_start max_applied_seq 2087065, open_ops 0
> 2012-12-20 22:18:21.016543 7fb9e57fa700 10 journal commit_start blocked, all open_ops have completed
> 2012-12-20 22:18:21.016545 7fb9e57fa700 10 journal commit_start nothing to do
> 2012-12-20 22:18:21.016548 7fb9e57fa700 10 journal commit_start
> 2012-12-20 22:18:21.016563 7fb9e57fa700 20 filestore(/srv/ceph/osd/osd.24) sync_entry waiting for max_interval 5.000000
> 2012-12-20 22:18:21.016563 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync done
> 2012-12-20 22:18:21.016570 7fb9e77fe700 10 filestore(/srv/ceph/osd/osd.24) sync_and_flush done
> 2012-12-20 22:18:21.016572 7fb9e77fe700 -1 osd.24 3473 removing pid file
> 2012-12-20 22:18:21.016665 7fb9e77fe700 -1 osd.24 3473 exit
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-20 15:28 ` Mark Nelson
@ 2012-12-20 15:51 ` Matthew Anderson
2012-12-20 15:59 ` Mark Nelson
0 siblings, 1 reply; 13+ messages in thread
From: Matthew Anderson @ 2012-12-20 15:51 UTC (permalink / raw)
To: 'Mark Nelson'; +Cc: ceph-devel@vger.kernel.org
Hi Mark,
Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
# ceph --admin-daemon ./ceph-osd.24.asok perf dump
{"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journal_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_bytes":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"sum":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":13354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":20868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1710232583,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing":0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":1708.284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000},"journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::apply_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_wip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op_latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out_bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000},"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcount":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"subop_push_in_bytes":0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"push":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"numpg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_from_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epoch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":{"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put_sum":795546573,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":199092,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"put_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
# ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
{ "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
"received_at": "2012-12-20 23:34:42.012131",
"age": "75.428051",
"duration": "0.009988",
"flag_point": "waiting for sub ops",
"client_info": { "client": "client.7636",
"tid": 101498},
"events": [
{ "time": "2012-12-20 23:34:42.012317",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:34:42.012425",
"event": "reached_pg"},
{ "time": "2012-12-20 23:34:42.012678",
"event": "started"},
{ "time": "2012-12-20 23:34:42.012720",
"event": "started"},
{ "time": "2012-12-20 23:34:42.012815",
"event": "sub_op_sent"},
{ "time": "2012-12-20 23:34:42.012958",
"event": "commit_queued_for_journal_write"},
{ "time": "2012-12-20 23:34:42.013033",
"event": "write_thread_in_journal_buffer"},
{ "time": "2012-12-20 23:34:42.013745",
"event": "journaled_completion_queued"},
{ "time": "2012-12-20 23:34:42.013803",
"event": "op_commit"},
{ "time": "2012-12-20 23:34:42.017414",
"event": "sub_op_commit_rec"},
{ "time": "2012-12-20 23:34:42.022066",
"event": "op_applied"},
{ "time": "2012-12-20 23:34:42.022119",
"event": "done"}]},
{ "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
"received_at": "2012-12-20 23:34:42.017048",
"age": "75.423134",
"duration": "0.000411",
"flag_point": "started",
"events": [
{ "time": "2012-12-20 23:34:42.017218",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:34:42.017363",
"event": "reached_pg"},
{ "time": "2012-12-20 23:34:42.017401",
"event": "started"},
{ "time": "2012-12-20 23:34:42.017459",
"event": "done"}]},
{ "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
"received_at": "2012-12-20 23:34:49.851021",
"age": "67.589161",
"duration": "0.007114",
"flag_point": "waiting for sub ops",
"client_info": { "client": "client.7636",
"tid": 101502},
"events": [
{ "time": "2012-12-20 23:34:49.851276",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:34:49.851429",
"event": "reached_pg"},
{ "time": "2012-12-20 23:34:49.851616",
"event": "started"},
{ "time": "2012-12-20 23:34:49.851648",
"event": "started"},
{ "time": "2012-12-20 23:34:49.851712",
"event": "sub_op_sent"},
{ "time": "2012-12-20 23:34:49.851770",
"event": "commit_queued_for_journal_write"},
{ "time": "2012-12-20 23:34:49.851915",
"event": "write_thread_in_journal_buffer"},
{ "time": "2012-12-20 23:34:49.853956",
"event": "journaled_completion_queued"},
{ "time": "2012-12-20 23:34:49.853992",
"event": "op_commit"},
{ "time": "2012-12-20 23:34:49.857016",
"event": "sub_op_commit_rec"},
{ "time": "2012-12-20 23:34:49.858077",
"event": "op_applied"},
{ "time": "2012-12-20 23:34:49.858135",
"event": "done"}]},
{ "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
"received_at": "2012-12-20 23:34:49.856762",
"age": "67.583420",
"duration": "0.000308",
"flag_point": "started",
"events": [
{ "time": "2012-12-20 23:34:49.856869",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:34:49.856969",
"event": "reached_pg"},
{ "time": "2012-12-20 23:34:49.857001",
"event": "started"},
{ "time": "2012-12-20 23:34:49.857070",
"event": "done"}]},
{ "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
"received_at": "2012-12-20 23:35:34.351518",
"age": "23.088664",
"duration": "0.009430",
"flag_point": "waiting for sub ops",
"client_info": { "client": "client.7636",
"tid": 101504},
"events": [
{ "time": "2012-12-20 23:35:34.351714",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:35:34.351811",
"event": "reached_pg"},
{ "time": "2012-12-20 23:35:34.352030",
"event": "started"},
{ "time": "2012-12-20 23:35:34.352071",
"event": "started"},
{ "time": "2012-12-20 23:35:34.352146",
"event": "sub_op_sent"},
{ "time": "2012-12-20 23:35:34.352213",
"event": "commit_queued_for_journal_write"},
{ "time": "2012-12-20 23:35:34.352287",
"event": "write_thread_in_journal_buffer"},
{ "time": "2012-12-20 23:35:34.355390",
"event": "journaled_completion_queued"},
{ "time": "2012-12-20 23:35:34.355475",
"event": "op_commit"},
{ "time": "2012-12-20 23:35:34.356661",
"event": "sub_op_commit_rec"},
{ "time": "2012-12-20 23:35:34.360894",
"event": "op_applied"},
{ "time": "2012-12-20 23:35:34.360948",
"event": "done"}]},
{ "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
"received_at": "2012-12-20 23:35:34.356286",
"age": "23.083896",
"duration": "0.000421",
"flag_point": "started",
"events": [
{ "time": "2012-12-20 23:35:34.356489",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:35:34.356598",
"event": "reached_pg"},
{ "time": "2012-12-20 23:35:34.356640",
"event": "started"},
{ "time": "2012-12-20 23:35:34.356707",
"event": "done"}]},
{ "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
"received_at": "2012-12-20 23:35:42.133103",
"age": "15.307079",
"duration": "0.008357",
"flag_point": "waiting for sub ops",
"client_info": { "client": "client.7636",
"tid": 101512},
"events": [
{ "time": "2012-12-20 23:35:42.133315",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:35:42.133427",
"event": "reached_pg"},
{ "time": "2012-12-20 23:35:42.133684",
"event": "started"},
{ "time": "2012-12-20 23:35:42.133723",
"event": "started"},
{ "time": "2012-12-20 23:35:42.133796",
"event": "sub_op_sent"},
{ "time": "2012-12-20 23:35:42.133908",
"event": "commit_queued_for_journal_write"},
{ "time": "2012-12-20 23:35:42.133995",
"event": "write_thread_in_journal_buffer"},
{ "time": "2012-12-20 23:35:42.136965",
"event": "journaled_completion_queued"},
{ "time": "2012-12-20 23:35:42.137025",
"event": "op_commit"},
{ "time": "2012-12-20 23:35:42.138456",
"event": "sub_op_commit_rec"},
{ "time": "2012-12-20 23:35:42.141408",
"event": "op_applied"},
{ "time": "2012-12-20 23:35:42.141460",
"event": "done"}]},
{ "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
"received_at": "2012-12-20 23:35:42.138087",
"age": "15.302095",
"duration": "0.000410",
"flag_point": "started",
"events": [
{ "time": "2012-12-20 23:35:42.138248",
"event": "waiting_for_osdmap"},
{ "time": "2012-12-20 23:35:42.138363",
"event": "reached_pg"},
{ "time": "2012-12-20 23:35:42.138443",
"event": "started"},
{ "time": "2012-12-20 23:35:42.138497",
"event": "done"}]}]}
# ceph --admin-daemon ./ceph-osd.24.asok config show
{ "name": "osd.24",
"cluster": "ceph",
"none": "0\/5",
"lockdep": "0\/1",
"context": "0\/1",
"crush": "1\/1",
"mds": "1\/5",
"mds_balancer": "1\/5",
"mds_locker": "1\/5",
"mds_log": "1\/5",
"mds_log_expire": "1\/5",
"mds_migrator": "1\/5",
"buffer": "0\/1",
"timer": "0\/1",
"filer": "0\/1",
"striper": "0\/1",
"objecter": "0\/1",
"rados": "0\/5",
"rbd": "0\/5",
"journaler": "0\/5",
"objectcacher": "0\/5",
"client": "0\/5",
"osd": "0\/5",
"optracker": "0\/5",
"objclass": "0\/5",
"filestore": "1\/3",
"journal": "1\/3",
"ms": "0\/5",
"mon": "1\/5",
"monc": "0\/10",
"paxos": "0\/5",
"tp": "0\/5",
"auth": "1\/5",
"crypto": "1\/5",
"finisher": "1\/1",
"heartbeatmap": "1\/5",
"perfcounter": "1\/5",
"rgw": "1\/5",
"hadoop": "1\/5",
"javaclient": "1\/5",
"asok": "1\/5",
"throttle": "1\/1",
"host": "KVM04",
"fsid": "00000000-0000-0000-0000-000000000000",
"public_addr": ":\/0",
"cluster_addr": ":\/0",
"public_network": "",
"cluster_network": "",
"num_client": "1",
"monmap": "",
"mon_host": "",
"lockdep": "false",
"admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
"daemonize": "true",
"pid_file": "\/var\/run\/ceph\/osd.24.pid",
"chdir": "\/",
"max_open_files": "0",
"fatal_signal_handlers": "true",
"log_file": "\/var\/log\/ceph\/osd.24.log",
"log_max_new": "1000",
"log_max_recent": "100000",
"log_to_stderr": "false",
"err_to_stderr": "true",
"log_to_syslog": "false",
"err_to_syslog": "false",
"log_flush_on_exit": "true",
"clog_to_monitors": "true",
"clog_to_syslog": "false",
"mon_cluster_log_to_syslog": "false",
"mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
"key": "",
"keyfile": "",
"keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
"heartbeat_interval": "5",
"heartbeat_file": "",
"perf": "true",
"ms_tcp_nodelay": "true",
"ms_initial_backoff": "0.2",
"ms_max_backoff": "15",
"ms_nocrc": "false",
"ms_die_on_bad_msg": "false",
"ms_dispatch_throttle_bytes": "104857600",
"ms_bind_ipv6": "false",
"ms_bind_port_min": "6800",
"ms_bind_port_max": "7100",
"ms_rwthread_stack_bytes": "1048576",
"ms_tcp_read_timeout": "900",
"ms_inject_socket_failures": "0",
"ms_inject_delay_type": "",
"ms_inject_delay_max": "1",
"ms_inject_delay_probability": "0",
"mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
"mon_initial_members": "",
"mon_sync_fs_threshold": "5",
"mon_tick_interval": "5",
"mon_subscribe_interval": "300",
"mon_osd_laggy_halflife": "3600",
"mon_osd_laggy_weight": "0.3",
"mon_osd_adjust_heartbeat_grace": "true",
"mon_osd_adjust_down_out_interval": "true",
"mon_osd_auto_mark_in": "false",
"mon_osd_auto_mark_auto_out_in": "true",
"mon_osd_auto_mark_new_in": "true",
"mon_osd_down_out_interval": "300",
"mon_osd_min_up_ratio": "0.3",
"mon_osd_min_in_ratio": "0.3",
"mon_lease": "5",
"mon_lease_renew_interval": "3",
"mon_lease_ack_timeout": "10",
"mon_clock_drift_allowed": "0.05",
"mon_clock_drift_warn_backoff": "5",
"mon_accept_timeout": "10",
"mon_pg_create_interval": "30",
"mon_pg_stuck_threshold": "300",
"mon_osd_full_ratio": "0.95",
"mon_osd_nearfull_ratio": "0.85",
"mon_globalid_prealloc": "100",
"mon_osd_report_timeout": "900",
"mon_force_standby_active": "true",
"mon_min_osdmap_epochs": "500",
"mon_max_pgmap_epochs": "500",
"mon_max_log_epochs": "500",
"mon_max_osd": "10000",
"mon_probe_timeout": "2",
"mon_slurp_timeout": "10",
"mon_slurp_bytes": "262144",
"mon_client_bytes": "104857600",
"mon_daemon_bytes": "419430400",
"mon_max_log_entries_per_event": "4096",
"paxos_max_join_drift": "10",
"paxos_propose_interval": "1",
"paxos_min_wait": "0.05",
"clock_offset": "0",
"auth_cluster_required": "none",
"auth_service_required": "none",
"auth_client_required": "none",
"auth_supported": "none",
"cephx_require_signatures": "false",
"cephx_sign_messages": "true",
"auth_mon_ticket_ttl": "43200",
"auth_service_ticket_ttl": "3600",
"mon_client_hunt_interval": "3",
"mon_client_ping_interval": "10",
"mon_client_max_log_entries_per_message": "1000",
"mon_max_pool_pg_num": "65536",
"client_cache_size": "16384",
"client_cache_mid": "0.75",
"client_use_random_mds": "false",
"client_mount_timeout": "30",
"client_tick_interval": "1",
"client_trace": "",
"client_readahead_min": "131072",
"client_readahead_max_bytes": "0",
"client_readahead_max_periods": "4",
"client_snapdir": ".snap",
"client_mountpoint": "\/",
"client_notify_timeout": "10",
"client_caps_release_delay": "5",
"client_oc": "true",
"client_oc_size": "209715200",
"client_oc_max_dirty": "104857600",
"client_oc_target_dirty": "8388608",
"client_oc_max_dirty_age": "5",
"client_oc_max_objects": "1000",
"client_debug_force_sync_read": "false",
"client_debug_inject_tick_delay": "0",
"fuse_use_invalidate_cb": "false",
"fuse_big_writes": "true",
"fuse_debug": "false",
"objecter_tick_interval": "5",
"objecter_timeout": "10",
"objecter_inflight_op_bytes": "104857600",
"objecter_inflight_ops": "1024",
"journaler_allow_split_entries": "true",
"journaler_write_head_interval": "15",
"journaler_prefetch_periods": "10",
"journaler_prezero_periods": "5",
"journaler_batch_interval": "0.001",
"journaler_batch_max": "0",
"mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
"mds_max_file_size": "1099511627776",
"mds_cache_size": "100000",
"mds_cache_mid": "0.7",
"mds_mem_max": "1048576",
"mds_dir_commit_ratio": "0.5",
"mds_dir_max_commit_size": "90",
"mds_decay_halflife": "5",
"mds_beacon_interval": "4",
"mds_beacon_grace": "15",
"mds_blacklist_interval": "1440",
"mds_session_timeout": "60",
"mds_session_autoclose": "300",
"mds_reconnect_timeout": "45",
"mds_tick_interval": "5",
"mds_dirstat_min_interval": "1",
"mds_scatter_nudge_interval": "5",
"mds_client_prealloc_inos": "1000",
"mds_early_reply": "true",
"mds_use_tmap": "true",
"mds_default_dir_hash": "2",
"mds_log": "true",
"mds_log_skip_corrupt_events": "false",
"mds_log_max_events": "-1",
"mds_log_max_segments": "30",
"mds_log_max_expiring": "20",
"mds_bal_sample_interval": "3",
"mds_bal_replicate_threshold": "8000",
"mds_bal_unreplicate_threshold": "0",
"mds_bal_frag": "false",
"mds_bal_split_size": "10000",
"mds_bal_split_rd": "25000",
"mds_bal_split_wr": "10000",
"mds_bal_split_bits": "3",
"mds_bal_merge_size": "50",
"mds_bal_merge_rd": "1000",
"mds_bal_merge_wr": "1000",
"mds_bal_interval": "10",
"mds_bal_fragment_interval": "5",
"mds_bal_idle_threshold": "0",
"mds_bal_max": "-1",
"mds_bal_max_until": "-1",
"mds_bal_mode": "0",
"mds_bal_min_rebalance": "0.1",
"mds_bal_min_start": "0.2",
"mds_bal_need_min": "0.8",
"mds_bal_need_max": "1.2",
"mds_bal_midchunk": "0.3",
"mds_bal_minchunk": "0.001",
"mds_bal_target_removal_min": "5",
"mds_bal_target_removal_max": "10",
"mds_replay_interval": "1",
"mds_shutdown_check": "0",
"mds_thrash_exports": "0",
"mds_thrash_fragments": "0",
"mds_dump_cache_on_map": "false",
"mds_dump_cache_after_rejoin": "false",
"mds_verify_scatter": "false",
"mds_debug_scatterstat": "false",
"mds_debug_frag": "false",
"mds_debug_auth_pins": "false",
"mds_debug_subtrees": "false",
"mds_kill_mdstable_at": "0",
"mds_kill_export_at": "0",
"mds_kill_import_at": "0",
"mds_kill_link_at": "0",
"mds_kill_rename_at": "0",
"mds_wipe_sessions": "false",
"mds_wipe_ino_prealloc": "false",
"mds_skip_ino": "0",
"max_mds": "1",
"mds_standby_for_name": "",
"mds_standby_for_rank": "-1",
"mds_standby_replay": "false",
"osd_auto_upgrade_tmap": "true",
"osd_tmapput_sets_uses_tmap": "false",
"osd_max_backfills": "10",
"osd_backfill_full_ratio": "0.85",
"osd_backfill_retry_interval": "10",
"osd_uuid": "00000000-0000-0000-0000-000000000000",
"osd_data": "\/srv\/ceph\/osd\/osd.24",
"osd_journal": "\/dev\/sda3",
"osd_journal_size": "5120",
"osd_max_write_size": "90",
"osd_max_pgls": "1024",
"osd_client_message_size_cap": "524288000",
"osd_pg_bits": "6",
"osd_pgp_bits": "6",
"osd_min_rep": "1",
"osd_max_rep": "10",
"osd_pool_default_crush_rule": "0",
"osd_pool_default_size": "2",
"osd_pool_default_min_size": "0",
"osd_pool_default_pg_num": "8",
"osd_pool_default_pgp_num": "8",
"osd_map_dedup": "true",
"osd_map_cache_size": "500",
"osd_map_message_max": "100",
"osd_op_threads": "2",
"osd_disk_threads": "1",
"osd_recovery_threads": "1",
"osd_recover_clone_overlap": "true",
"osd_backfill_scan_min": "64",
"osd_backfill_scan_max": "512",
"osd_op_thread_timeout": "30",
"osd_recovery_thread_timeout": "30",
"osd_snap_trim_thread_timeout": "3600",
"osd_scrub_thread_timeout": "60",
"osd_scrub_finalize_thread_timeout": "600",
"osd_remove_thread_timeout": "3600",
"osd_command_thread_timeout": "600",
"osd_age": "0.8",
"osd_age_time": "0",
"osd_heartbeat_addr": ":\/0",
"osd_heartbeat_interval": "6",
"osd_heartbeat_grace": "20",
"osd_mon_heartbeat_interval": "30",
"osd_mon_report_interval_max": "120",
"osd_mon_report_interval_min": "5",
"osd_mon_ack_timeout": "30",
"osd_min_down_reporters": "1",
"osd_min_down_reports": "3",
"osd_default_data_pool_replay_window": "45",
"osd_preserve_trimmed_log": "false",
"osd_auto_mark_unfound_lost": "false",
"osd_recovery_delay_start": "0",
"osd_recovery_max_active": "50",
"osd_recovery_max_chunk": "8388608",
"osd_recovery_forget_lost_objects": "false",
"osd_max_scrubs": "1",
"osd_scrub_load_threshold": "0.5",
"osd_scrub_min_interval": "300",
"osd_scrub_max_interval": "86400",
"osd_deep_scrub_interval": "604800",
"osd_deep_scrub_stride": "524288",
"osd_auto_weight": "false",
"osd_class_dir": "\/usr\/local\/lib\/rados-classes",
"osd_check_for_log_corruption": "false",
"osd_use_stale_snap": "false",
"osd_rollback_to_cluster_snap": "",
"osd_default_notify_timeout": "30",
"osd_kill_backfill_at": "0",
"osd_min_pg_log_entries": "1000",
"osd_op_complaint_time": "30",
"osd_command_max_records": "256",
"osd_op_log_threshold": "5",
"osd_verify_sparse_read_holes": "false",
"osd_debug_drop_ping_probability": "0",
"osd_debug_drop_ping_duration": "0",
"osd_debug_drop_pg_create_probability": "0",
"osd_debug_drop_pg_create_duration": "1",
"osd_debug_drop_op_probability": "0",
"osd_op_history_size": "20",
"osd_op_history_duration": "600",
"osd_target_transaction_size": "300",
"osd_client_op_priority": "63",
"osd_recovery_op_priority": "10",
"filestore": "false",
"filestore_index_retry_probability": "0",
"filestore_debug_omap_check": "false",
"filestore_xattr_use_omap": "false",
"filestore_max_inline_xattr_size": "512",
"filestore_max_inline_xattrs": "2",
"filestore_max_sync_interval": "5",
"filestore_min_sync_interval": "0.01",
"filestore_btrfs_snap": "true",
"filestore_btrfs_clone_range": "true",
"filestore_fsync_flushes_journal_data": "false",
"filestore_fiemap": "false",
"filestore_flusher": "true",
"filestore_flusher_max_fds": "512",
"filestore_flush_min": "65536",
"filestore_sync_flush": "false",
"filestore_journal_parallel": "false",
"filestore_journal_writeahead": "false",
"filestore_journal_trailing": "false",
"filestore_queue_max_ops": "500",
"filestore_queue_max_bytes": "104857600",
"filestore_queue_committing_max_ops": "500",
"filestore_queue_committing_max_bytes": "104857600",
"filestore_op_threads": "2",
"filestore_op_thread_timeout": "60",
"filestore_op_thread_suicide_timeout": "180",
"filestore_commit_timeout": "600",
"filestore_fiemap_threshold": "4096",
"filestore_merge_threshold": "10",
"filestore_split_multiple": "2",
"filestore_update_to": "1000",
"filestore_blackhole": "false",
"filestore_dump_file": "",
"filestore_kill_at": "0",
"filestore_fail_eio": "true",
"journal_dio": "true",
"journal_aio": "false",
"journal_block_align": "true",
"journal_max_write_bytes": "10485760",
"journal_max_write_entries": "100",
"journal_queue_max_ops": "500",
"journal_queue_max_bytes": "104857600",
"journal_align_min_size": "65536",
"journal_replay_from": "0",
"journal_zero_on_create": "false",
"rbd_cache": "false",
"rbd_cache_size": "33554432",
"rbd_cache_max_dirty": "25165824",
"rbd_cache_target_dirty": "16777216",
"rbd_cache_max_dirty_age": "1",
"nss_db_path": "",
"rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
"rgw_enable_apis": "s3, swift, swift_auth, admin",
"rgw_cache_enabled": "true",
"rgw_cache_lru_size": "10000",
"rgw_socket_path": "",
"rgw_dns_name": "",
"rgw_swift_url": "",
"rgw_swift_url_prefix": "swift",
"rgw_swift_auth_url": "",
"rgw_swift_auth_entry": "auth",
"rgw_keystone_url": "",
"rgw_keystone_admin_token": "",
"rgw_keystone_accepted_roles": "Member, admin",
"rgw_keystone_token_cache_size": "10000",
"rgw_keystone_revocation_interval": "900",
"rgw_admin_entry": "admin",
"rgw_enforce_swift_acls": "true",
"rgw_swift_token_expiration": "86400",
"rgw_print_continue": "true",
"rgw_remote_addr_param": "REMOTE_ADDR",
"rgw_op_thread_timeout": "600",
"rgw_op_thread_suicide_timeout": "0",
"rgw_thread_pool_size": "100",
"rgw_num_control_oids": "8",
"rgw_cluster_root_pool": ".rgw.root",
"rgw_log_nonexistent_bucket": "false",
"rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
"rgw_log_object_name_utc": "false",
"rgw_usage_max_shards": "32",
"rgw_usage_max_user_shards": "1",
"rgw_enable_ops_log": "true",
"rgw_enable_usage_log": "true",
"rgw_ops_log_rados": "true",
"rgw_ops_log_socket_path": "",
"rgw_ops_log_data_backlog": "5242880",
"rgw_usage_log_flush_threshold": "1024",
"rgw_usage_log_tick_interval": "30",
"rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
"rgw_intent_log_object_name_utc": "false",
"rgw_init_timeout": "30",
"rgw_mime_types_file": "\/etc\/mime.types",
"rgw_gc_max_objs": "32",
"rgw_gc_obj_min_wait": "7200",
"rgw_gc_processor_max_time": "3600",
"rgw_gc_processor_period": "3600",
"rgw_s3_success_create_obj_status": "0",
"rgw_resolve_cname": "false",
"rgw_obj_stripe_size": "4194304",
"rgw_extended_http_attrs": "",
"mutex_perf_counter": "false",
"internal_safe_to_start_threads": "true"}
-----Original Message-----
From: Mark Nelson [mailto:mark.nelson@inktank.com]
Sent: Thursday, 20 December 2012 11:28 PM
To: Matthew Anderson
Cc: ceph-devel@vger.kernel.org
Subject: Re: OSD's slow down to a crawl
Hi Matt,
Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
Sebastien Han made a nice little blog entry describing it:
http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
You can get a lot of information about the current state of things.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: OSD's slow down to a crawl
2012-12-20 15:51 ` Matthew Anderson
@ 2012-12-20 15:59 ` Mark Nelson
2012-12-20 16:08 ` Matthew Anderson
0 siblings, 1 reply; 13+ messages in thread
From: Mark Nelson @ 2012-12-20 15:59 UTC (permalink / raw)
To: Matthew Anderson; +Cc: ceph-devel@vger.kernel.org
Out of curiosity, if you fire up a rados bench instance on one of the
nodes with say, 256 concurrent writes, do any of the writes complete?
Mark
On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> Hi Mark,
>
> Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
>
> It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
>
> The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> # ceph --admin-daemon ./ceph-osd.24.asok perf dump
>
> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journal_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_bytes":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"sum":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":13354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":20868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1710232583,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing":0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":1708.284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000},"journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait":{"avgcount":0,"sum":0.00000
0000}},"mutex-FileJournal::write_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::apply_lock":{"wai
t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_wip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op_latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out_bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000},"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcount":5700,"sum":26.877085000},"s
ubop_w":0,"subop_w_in_bytes":813224156,"subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"subop_push_in_bytes"
:0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"push":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"numpg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_from_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epoch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgcount":0,"sum":0.000000000}},"t
hrottle-msgr_dispatch_throttler-client":{"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put_sum":795546573,"wa
it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":199092,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes":{"val":0,"max":524288000,"ge
t":31308,"get_sum":795353720,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"put_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
>
> # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
>
> { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> "received_at": "2012-12-20 23:34:42.012131",
> "age": "75.428051",
> "duration": "0.009988",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101498},
> "events": [
> { "time": "2012-12-20 23:34:42.012317",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:42.012425",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:42.012678",
> "event": "started"},
> { "time": "2012-12-20 23:34:42.012720",
> "event": "started"},
> { "time": "2012-12-20 23:34:42.012815",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:34:42.012958",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:34:42.013033",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:34:42.013745",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:34:42.013803",
> "event": "op_commit"},
> { "time": "2012-12-20 23:34:42.017414",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:34:42.022066",
> "event": "op_applied"},
> { "time": "2012-12-20 23:34:42.022119",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:34:42.017048",
> "age": "75.423134",
> "duration": "0.000411",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:34:42.017218",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:42.017363",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:42.017401",
> "event": "started"},
> { "time": "2012-12-20 23:34:42.017459",
> "event": "done"}]},
> { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> "received_at": "2012-12-20 23:34:49.851021",
> "age": "67.589161",
> "duration": "0.007114",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101502},
> "events": [
> { "time": "2012-12-20 23:34:49.851276",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:49.851429",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:49.851616",
> "event": "started"},
> { "time": "2012-12-20 23:34:49.851648",
> "event": "started"},
> { "time": "2012-12-20 23:34:49.851712",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:34:49.851770",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:34:49.851915",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:34:49.853956",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:34:49.853992",
> "event": "op_commit"},
> { "time": "2012-12-20 23:34:49.857016",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:34:49.858077",
> "event": "op_applied"},
> { "time": "2012-12-20 23:34:49.858135",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:34:49.856762",
> "age": "67.583420",
> "duration": "0.000308",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:34:49.856869",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:49.856969",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:49.857001",
> "event": "started"},
> { "time": "2012-12-20 23:34:49.857070",
> "event": "done"}]},
> { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> "received_at": "2012-12-20 23:35:34.351518",
> "age": "23.088664",
> "duration": "0.009430",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101504},
> "events": [
> { "time": "2012-12-20 23:35:34.351714",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:34.351811",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:34.352030",
> "event": "started"},
> { "time": "2012-12-20 23:35:34.352071",
> "event": "started"},
> { "time": "2012-12-20 23:35:34.352146",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:35:34.352213",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:35:34.352287",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:35:34.355390",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:35:34.355475",
> "event": "op_commit"},
> { "time": "2012-12-20 23:35:34.356661",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:35:34.360894",
> "event": "op_applied"},
> { "time": "2012-12-20 23:35:34.360948",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:35:34.356286",
> "age": "23.083896",
> "duration": "0.000421",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:35:34.356489",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:34.356598",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:34.356640",
> "event": "started"},
> { "time": "2012-12-20 23:35:34.356707",
> "event": "done"}]},
> { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> "received_at": "2012-12-20 23:35:42.133103",
> "age": "15.307079",
> "duration": "0.008357",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101512},
> "events": [
> { "time": "2012-12-20 23:35:42.133315",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:42.133427",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:42.133684",
> "event": "started"},
> { "time": "2012-12-20 23:35:42.133723",
> "event": "started"},
> { "time": "2012-12-20 23:35:42.133796",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:35:42.133908",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:35:42.133995",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:35:42.136965",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:35:42.137025",
> "event": "op_commit"},
> { "time": "2012-12-20 23:35:42.138456",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:35:42.141408",
> "event": "op_applied"},
> { "time": "2012-12-20 23:35:42.141460",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:35:42.138087",
> "age": "15.302095",
> "duration": "0.000410",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:35:42.138248",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:42.138363",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:42.138443",
> "event": "started"},
> { "time": "2012-12-20 23:35:42.138497",
> "event": "done"}]}]}
>
> # ceph --admin-daemon ./ceph-osd.24.asok config show
> { "name": "osd.24",
> "cluster": "ceph",
> "none": "0\/5",
> "lockdep": "0\/1",
> "context": "0\/1",
> "crush": "1\/1",
> "mds": "1\/5",
> "mds_balancer": "1\/5",
> "mds_locker": "1\/5",
> "mds_log": "1\/5",
> "mds_log_expire": "1\/5",
> "mds_migrator": "1\/5",
> "buffer": "0\/1",
> "timer": "0\/1",
> "filer": "0\/1",
> "striper": "0\/1",
> "objecter": "0\/1",
> "rados": "0\/5",
> "rbd": "0\/5",
> "journaler": "0\/5",
> "objectcacher": "0\/5",
> "client": "0\/5",
> "osd": "0\/5",
> "optracker": "0\/5",
> "objclass": "0\/5",
> "filestore": "1\/3",
> "journal": "1\/3",
> "ms": "0\/5",
> "mon": "1\/5",
> "monc": "0\/10",
> "paxos": "0\/5",
> "tp": "0\/5",
> "auth": "1\/5",
> "crypto": "1\/5",
> "finisher": "1\/1",
> "heartbeatmap": "1\/5",
> "perfcounter": "1\/5",
> "rgw": "1\/5",
> "hadoop": "1\/5",
> "javaclient": "1\/5",
> "asok": "1\/5",
> "throttle": "1\/1",
> "host": "KVM04",
> "fsid": "00000000-0000-0000-0000-000000000000",
> "public_addr": ":\/0",
> "cluster_addr": ":\/0",
> "public_network": "",
> "cluster_network": "",
> "num_client": "1",
> "monmap": "",
> "mon_host": "",
> "lockdep": "false",
> "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> "daemonize": "true",
> "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> "chdir": "\/",
> "max_open_files": "0",
> "fatal_signal_handlers": "true",
> "log_file": "\/var\/log\/ceph\/osd.24.log",
> "log_max_new": "1000",
> "log_max_recent": "100000",
> "log_to_stderr": "false",
> "err_to_stderr": "true",
> "log_to_syslog": "false",
> "err_to_syslog": "false",
> "log_flush_on_exit": "true",
> "clog_to_monitors": "true",
> "clog_to_syslog": "false",
> "mon_cluster_log_to_syslog": "false",
> "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> "key": "",
> "keyfile": "",
> "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> "heartbeat_interval": "5",
> "heartbeat_file": "",
> "perf": "true",
> "ms_tcp_nodelay": "true",
> "ms_initial_backoff": "0.2",
> "ms_max_backoff": "15",
> "ms_nocrc": "false",
> "ms_die_on_bad_msg": "false",
> "ms_dispatch_throttle_bytes": "104857600",
> "ms_bind_ipv6": "false",
> "ms_bind_port_min": "6800",
> "ms_bind_port_max": "7100",
> "ms_rwthread_stack_bytes": "1048576",
> "ms_tcp_read_timeout": "900",
> "ms_inject_socket_failures": "0",
> "ms_inject_delay_type": "",
> "ms_inject_delay_max": "1",
> "ms_inject_delay_probability": "0",
> "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> "mon_initial_members": "",
> "mon_sync_fs_threshold": "5",
> "mon_tick_interval": "5",
> "mon_subscribe_interval": "300",
> "mon_osd_laggy_halflife": "3600",
> "mon_osd_laggy_weight": "0.3",
> "mon_osd_adjust_heartbeat_grace": "true",
> "mon_osd_adjust_down_out_interval": "true",
> "mon_osd_auto_mark_in": "false",
> "mon_osd_auto_mark_auto_out_in": "true",
> "mon_osd_auto_mark_new_in": "true",
> "mon_osd_down_out_interval": "300",
> "mon_osd_min_up_ratio": "0.3",
> "mon_osd_min_in_ratio": "0.3",
> "mon_lease": "5",
> "mon_lease_renew_interval": "3",
> "mon_lease_ack_timeout": "10",
> "mon_clock_drift_allowed": "0.05",
> "mon_clock_drift_warn_backoff": "5",
> "mon_accept_timeout": "10",
> "mon_pg_create_interval": "30",
> "mon_pg_stuck_threshold": "300",
> "mon_osd_full_ratio": "0.95",
> "mon_osd_nearfull_ratio": "0.85",
> "mon_globalid_prealloc": "100",
> "mon_osd_report_timeout": "900",
> "mon_force_standby_active": "true",
> "mon_min_osdmap_epochs": "500",
> "mon_max_pgmap_epochs": "500",
> "mon_max_log_epochs": "500",
> "mon_max_osd": "10000",
> "mon_probe_timeout": "2",
> "mon_slurp_timeout": "10",
> "mon_slurp_bytes": "262144",
> "mon_client_bytes": "104857600",
> "mon_daemon_bytes": "419430400",
> "mon_max_log_entries_per_event": "4096",
> "paxos_max_join_drift": "10",
> "paxos_propose_interval": "1",
> "paxos_min_wait": "0.05",
> "clock_offset": "0",
> "auth_cluster_required": "none",
> "auth_service_required": "none",
> "auth_client_required": "none",
> "auth_supported": "none",
> "cephx_require_signatures": "false",
> "cephx_sign_messages": "true",
> "auth_mon_ticket_ttl": "43200",
> "auth_service_ticket_ttl": "3600",
> "mon_client_hunt_interval": "3",
> "mon_client_ping_interval": "10",
> "mon_client_max_log_entries_per_message": "1000",
> "mon_max_pool_pg_num": "65536",
> "client_cache_size": "16384",
> "client_cache_mid": "0.75",
> "client_use_random_mds": "false",
> "client_mount_timeout": "30",
> "client_tick_interval": "1",
> "client_trace": "",
> "client_readahead_min": "131072",
> "client_readahead_max_bytes": "0",
> "client_readahead_max_periods": "4",
> "client_snapdir": ".snap",
> "client_mountpoint": "\/",
> "client_notify_timeout": "10",
> "client_caps_release_delay": "5",
> "client_oc": "true",
> "client_oc_size": "209715200",
> "client_oc_max_dirty": "104857600",
> "client_oc_target_dirty": "8388608",
> "client_oc_max_dirty_age": "5",
> "client_oc_max_objects": "1000",
> "client_debug_force_sync_read": "false",
> "client_debug_inject_tick_delay": "0",
> "fuse_use_invalidate_cb": "false",
> "fuse_big_writes": "true",
> "fuse_debug": "false",
> "objecter_tick_interval": "5",
> "objecter_timeout": "10",
> "objecter_inflight_op_bytes": "104857600",
> "objecter_inflight_ops": "1024",
> "journaler_allow_split_entries": "true",
> "journaler_write_head_interval": "15",
> "journaler_prefetch_periods": "10",
> "journaler_prezero_periods": "5",
> "journaler_batch_interval": "0.001",
> "journaler_batch_max": "0",
> "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> "mds_max_file_size": "1099511627776",
> "mds_cache_size": "100000",
> "mds_cache_mid": "0.7",
> "mds_mem_max": "1048576",
> "mds_dir_commit_ratio": "0.5",
> "mds_dir_max_commit_size": "90",
> "mds_decay_halflife": "5",
> "mds_beacon_interval": "4",
> "mds_beacon_grace": "15",
> "mds_blacklist_interval": "1440",
> "mds_session_timeout": "60",
> "mds_session_autoclose": "300",
> "mds_reconnect_timeout": "45",
> "mds_tick_interval": "5",
> "mds_dirstat_min_interval": "1",
> "mds_scatter_nudge_interval": "5",
> "mds_client_prealloc_inos": "1000",
> "mds_early_reply": "true",
> "mds_use_tmap": "true",
> "mds_default_dir_hash": "2",
> "mds_log": "true",
> "mds_log_skip_corrupt_events": "false",
> "mds_log_max_events": "-1",
> "mds_log_max_segments": "30",
> "mds_log_max_expiring": "20",
> "mds_bal_sample_interval": "3",
> "mds_bal_replicate_threshold": "8000",
> "mds_bal_unreplicate_threshold": "0",
> "mds_bal_frag": "false",
> "mds_bal_split_size": "10000",
> "mds_bal_split_rd": "25000",
> "mds_bal_split_wr": "10000",
> "mds_bal_split_bits": "3",
> "mds_bal_merge_size": "50",
> "mds_bal_merge_rd": "1000",
> "mds_bal_merge_wr": "1000",
> "mds_bal_interval": "10",
> "mds_bal_fragment_interval": "5",
> "mds_bal_idle_threshold": "0",
> "mds_bal_max": "-1",
> "mds_bal_max_until": "-1",
> "mds_bal_mode": "0",
> "mds_bal_min_rebalance": "0.1",
> "mds_bal_min_start": "0.2",
> "mds_bal_need_min": "0.8",
> "mds_bal_need_max": "1.2",
> "mds_bal_midchunk": "0.3",
> "mds_bal_minchunk": "0.001",
> "mds_bal_target_removal_min": "5",
> "mds_bal_target_removal_max": "10",
> "mds_replay_interval": "1",
> "mds_shutdown_check": "0",
> "mds_thrash_exports": "0",
> "mds_thrash_fragments": "0",
> "mds_dump_cache_on_map": "false",
> "mds_dump_cache_after_rejoin": "false",
> "mds_verify_scatter": "false",
> "mds_debug_scatterstat": "false",
> "mds_debug_frag": "false",
> "mds_debug_auth_pins": "false",
> "mds_debug_subtrees": "false",
> "mds_kill_mdstable_at": "0",
> "mds_kill_export_at": "0",
> "mds_kill_import_at": "0",
> "mds_kill_link_at": "0",
> "mds_kill_rename_at": "0",
> "mds_wipe_sessions": "false",
> "mds_wipe_ino_prealloc": "false",
> "mds_skip_ino": "0",
> "max_mds": "1",
> "mds_standby_for_name": "",
> "mds_standby_for_rank": "-1",
> "mds_standby_replay": "false",
> "osd_auto_upgrade_tmap": "true",
> "osd_tmapput_sets_uses_tmap": "false",
> "osd_max_backfills": "10",
> "osd_backfill_full_ratio": "0.85",
> "osd_backfill_retry_interval": "10",
> "osd_uuid": "00000000-0000-0000-0000-000000000000",
> "osd_data": "\/srv\/ceph\/osd\/osd.24",
> "osd_journal": "\/dev\/sda3",
> "osd_journal_size": "5120",
> "osd_max_write_size": "90",
> "osd_max_pgls": "1024",
> "osd_client_message_size_cap": "524288000",
> "osd_pg_bits": "6",
> "osd_pgp_bits": "6",
> "osd_min_rep": "1",
> "osd_max_rep": "10",
> "osd_pool_default_crush_rule": "0",
> "osd_pool_default_size": "2",
> "osd_pool_default_min_size": "0",
> "osd_pool_default_pg_num": "8",
> "osd_pool_default_pgp_num": "8",
> "osd_map_dedup": "true",
> "osd_map_cache_size": "500",
> "osd_map_message_max": "100",
> "osd_op_threads": "2",
> "osd_disk_threads": "1",
> "osd_recovery_threads": "1",
> "osd_recover_clone_overlap": "true",
> "osd_backfill_scan_min": "64",
> "osd_backfill_scan_max": "512",
> "osd_op_thread_timeout": "30",
> "osd_recovery_thread_timeout": "30",
> "osd_snap_trim_thread_timeout": "3600",
> "osd_scrub_thread_timeout": "60",
> "osd_scrub_finalize_thread_timeout": "600",
> "osd_remove_thread_timeout": "3600",
> "osd_command_thread_timeout": "600",
> "osd_age": "0.8",
> "osd_age_time": "0",
> "osd_heartbeat_addr": ":\/0",
> "osd_heartbeat_interval": "6",
> "osd_heartbeat_grace": "20",
> "osd_mon_heartbeat_interval": "30",
> "osd_mon_report_interval_max": "120",
> "osd_mon_report_interval_min": "5",
> "osd_mon_ack_timeout": "30",
> "osd_min_down_reporters": "1",
> "osd_min_down_reports": "3",
> "osd_default_data_pool_replay_window": "45",
> "osd_preserve_trimmed_log": "false",
> "osd_auto_mark_unfound_lost": "false",
> "osd_recovery_delay_start": "0",
> "osd_recovery_max_active": "50",
> "osd_recovery_max_chunk": "8388608",
> "osd_recovery_forget_lost_objects": "false",
> "osd_max_scrubs": "1",
> "osd_scrub_load_threshold": "0.5",
> "osd_scrub_min_interval": "300",
> "osd_scrub_max_interval": "86400",
> "osd_deep_scrub_interval": "604800",
> "osd_deep_scrub_stride": "524288",
> "osd_auto_weight": "false",
> "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> "osd_check_for_log_corruption": "false",
> "osd_use_stale_snap": "false",
> "osd_rollback_to_cluster_snap": "",
> "osd_default_notify_timeout": "30",
> "osd_kill_backfill_at": "0",
> "osd_min_pg_log_entries": "1000",
> "osd_op_complaint_time": "30",
> "osd_command_max_records": "256",
> "osd_op_log_threshold": "5",
> "osd_verify_sparse_read_holes": "false",
> "osd_debug_drop_ping_probability": "0",
> "osd_debug_drop_ping_duration": "0",
> "osd_debug_drop_pg_create_probability": "0",
> "osd_debug_drop_pg_create_duration": "1",
> "osd_debug_drop_op_probability": "0",
> "osd_op_history_size": "20",
> "osd_op_history_duration": "600",
> "osd_target_transaction_size": "300",
> "osd_client_op_priority": "63",
> "osd_recovery_op_priority": "10",
> "filestore": "false",
> "filestore_index_retry_probability": "0",
> "filestore_debug_omap_check": "false",
> "filestore_xattr_use_omap": "false",
> "filestore_max_inline_xattr_size": "512",
> "filestore_max_inline_xattrs": "2",
> "filestore_max_sync_interval": "5",
> "filestore_min_sync_interval": "0.01",
> "filestore_btrfs_snap": "true",
> "filestore_btrfs_clone_range": "true",
> "filestore_fsync_flushes_journal_data": "false",
> "filestore_fiemap": "false",
> "filestore_flusher": "true",
> "filestore_flusher_max_fds": "512",
> "filestore_flush_min": "65536",
> "filestore_sync_flush": "false",
> "filestore_journal_parallel": "false",
> "filestore_journal_writeahead": "false",
> "filestore_journal_trailing": "false",
> "filestore_queue_max_ops": "500",
> "filestore_queue_max_bytes": "104857600",
> "filestore_queue_committing_max_ops": "500",
> "filestore_queue_committing_max_bytes": "104857600",
> "filestore_op_threads": "2",
> "filestore_op_thread_timeout": "60",
> "filestore_op_thread_suicide_timeout": "180",
> "filestore_commit_timeout": "600",
> "filestore_fiemap_threshold": "4096",
> "filestore_merge_threshold": "10",
> "filestore_split_multiple": "2",
> "filestore_update_to": "1000",
> "filestore_blackhole": "false",
> "filestore_dump_file": "",
> "filestore_kill_at": "0",
> "filestore_fail_eio": "true",
> "journal_dio": "true",
> "journal_aio": "false",
> "journal_block_align": "true",
> "journal_max_write_bytes": "10485760",
> "journal_max_write_entries": "100",
> "journal_queue_max_ops": "500",
> "journal_queue_max_bytes": "104857600",
> "journal_align_min_size": "65536",
> "journal_replay_from": "0",
> "journal_zero_on_create": "false",
> "rbd_cache": "false",
> "rbd_cache_size": "33554432",
> "rbd_cache_max_dirty": "25165824",
> "rbd_cache_target_dirty": "16777216",
> "rbd_cache_max_dirty_age": "1",
> "nss_db_path": "",
> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> "rgw_enable_apis": "s3, swift, swift_auth, admin",
> "rgw_cache_enabled": "true",
> "rgw_cache_lru_size": "10000",
> "rgw_socket_path": "",
> "rgw_dns_name": "",
> "rgw_swift_url": "",
> "rgw_swift_url_prefix": "swift",
> "rgw_swift_auth_url": "",
> "rgw_swift_auth_entry": "auth",
> "rgw_keystone_url": "",
> "rgw_keystone_admin_token": "",
> "rgw_keystone_accepted_roles": "Member, admin",
> "rgw_keystone_token_cache_size": "10000",
> "rgw_keystone_revocation_interval": "900",
> "rgw_admin_entry": "admin",
> "rgw_enforce_swift_acls": "true",
> "rgw_swift_token_expiration": "86400",
> "rgw_print_continue": "true",
> "rgw_remote_addr_param": "REMOTE_ADDR",
> "rgw_op_thread_timeout": "600",
> "rgw_op_thread_suicide_timeout": "0",
> "rgw_thread_pool_size": "100",
> "rgw_num_control_oids": "8",
> "rgw_cluster_root_pool": ".rgw.root",
> "rgw_log_nonexistent_bucket": "false",
> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> "rgw_log_object_name_utc": "false",
> "rgw_usage_max_shards": "32",
> "rgw_usage_max_user_shards": "1",
> "rgw_enable_ops_log": "true",
> "rgw_enable_usage_log": "true",
> "rgw_ops_log_rados": "true",
> "rgw_ops_log_socket_path": "",
> "rgw_ops_log_data_backlog": "5242880",
> "rgw_usage_log_flush_threshold": "1024",
> "rgw_usage_log_tick_interval": "30",
> "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> "rgw_intent_log_object_name_utc": "false",
> "rgw_init_timeout": "30",
> "rgw_mime_types_file": "\/etc\/mime.types",
> "rgw_gc_max_objs": "32",
> "rgw_gc_obj_min_wait": "7200",
> "rgw_gc_processor_max_time": "3600",
> "rgw_gc_processor_period": "3600",
> "rgw_s3_success_create_obj_status": "0",
> "rgw_resolve_cname": "false",
> "rgw_obj_stripe_size": "4194304",
> "rgw_extended_http_attrs": "",
> "mutex_perf_counter": "false",
> "internal_safe_to_start_threads": "true"}
>
>
> -----Original Message-----
> From: Mark Nelson [mailto:mark.nelson@inktank.com]
> Sent: Thursday, 20 December 2012 11:28 PM
> To: Matthew Anderson
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: OSD's slow down to a crawl
>
> Hi Matt,
>
> Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> Sebastien Han made a nice little blog entry describing it:
>
> http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
>
> You can get a lot of information about the current state of things.
>
> Mark
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-20 15:59 ` Mark Nelson
@ 2012-12-20 16:08 ` Matthew Anderson
2012-12-20 16:22 ` Mark Nelson
2012-12-20 16:30 ` Sage Weil
0 siblings, 2 replies; 13+ messages in thread
From: Matthew Anderson @ 2012-12-20 16:08 UTC (permalink / raw)
To: 'Mark Nelson'; +Cc: ceph-devel@vger.kernel.org
# rados bench 60 write -t 256 -p ssd
Maintaining 256 concurrent writes of 4194304 bytes for at least 60 seconds.
Object prefix: benchmark_data_KVM03_12985
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 0 0 0 0 0 - 0
1 45 45 0 0 0 - 0
2 71 71 0 0 0 - 0
3 90 90 0 0 0 - 0
4 125 125 0 0 0 - 0
5 136 136 0 0 0 - 0
6 144 144 0 0 0 - 0
7 175 175 0 0 0 - 0
8 201 201 0 0 0 - 0
9 217 217 0 0 0 - 0
10 228 228 0 0 0 - 0
11 251 251 0 0 0 - 0
12 255 266 11 3.66581 3.66667 11.7027 11.6013
13 255 271 16 4.92194 20 12.9795 11.9979
14 255 291 36 10.2833 80 13.1795 12.6478
15 255 301 46 12.2639 40 13.9424 12.9123
16 255 303 48 11.9973 8 14.0277 12.9579
17 255 321 66 15.526 72 15.1408 13.4616
18 255 345 90 19.9956 96 15.5673 13.9937
19 255 349 94 19.7851 16 15.707 14.0635
2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 255 361 106 21.1953 48 16.4053 14.3121
21 255 367 112 21.3285 24 17.4263 14.4795
22 255 370 115 20.9043 12 17.3579 14.5544
23 255 387 132 22.9513 68 18.3185 15.0867
24 255 394 139 23.1614 28 18.2868 15.2504
25 255 404 149 23.8346 40 18.741 15.5048
26 255 410 155 23.8407 24 19.032 15.6358
27 255 410 155 22.9577 0 - 15.6358
28 255 410 155 22.1378 0 - 15.6358
29 255 411 156 21.5124 1.33333 22.6927 15.6811
30 255 431 176 23.4614 80 22.6577 16.4822
31 255 447 192 24.7687 64 23.2345 17.0563
32 255 464 209 26.1192 68 23.3788 17.5859
33 255 485 230 27.8726 84 22.8924 18.1122
34 255 497 242 28.4642 48 23.563 18.3693
35 255 508 253 28.9078 44 23.0516 18.5935
36 255 514 259 28.7713 24 24.4415 18.7278
37 255 530 275 29.723 64 23.847 19.0424
38 255 541 286 30.0985 44 24.5757 19.2427
39 255 544 289 29.6344 12 24.6802 19.2984
2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
40 255 544 289 28.8935 0 - 19.2984
41 255 544 289 28.1888 0 - 19.2984
42 255 544 289 27.5176 0 - 19.2984
43 255 544 289 26.8776 0 - 19.2984
44 255 544 289 26.2668 0 - 19.2984
45 255 544 289 25.6831 0 - 19.2984
46 255 544 289 25.1248 0 - 19.2984
47 255 544 289 24.5902 0 - 19.2984
48 255 544 289 24.0779 0 - 19.2984
49 255 544 289 23.5865 0 - 19.2984
50 255 544 289 23.1148 0 - 19.2984
51 255 544 289 22.6616 0 - 19.2984
52 255 544 289 22.2258 0 - 19.2984
53 255 544 289 21.8065 0 - 19.2984
54 255 544 289 21.4026 0 - 19.2984
55 255 544 289 21.0135 0 - 19.2984
56 255 544 289 20.6383 0 - 19.2984
57 255 544 289 20.2762 0 - 19.2984
58 255 544 289 19.9266 0 - 19.2984
59 255 544 289 19.5889 0 - 19.2984
2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
60 255 544 289 19.2624 0 - 19.2984
61 255 544 289 18.9466 0 - 19.2984
62 255 544 289 18.641 0 - 19.2984
63 255 544 289 18.3452 0 - 19.2984
64 255 544 289 18.0585 0 - 19.2984
65 255 544 289 17.7807 0 - 19.2984
66 255 544 289 17.5113 0 - 19.2984
67 133 545 412 24.5916 17.5714 37.3574 27.4834
Total time run: 68.000740
Total writes made: 545
Write size: 4194304
Bandwidth (MB/sec): 32.058
Stddev Bandwidth: 26.8837
Max bandwidth (MB/sec): 96
Min bandwidth (MB/sec): 0
Average Latency: 29.2801
Stddev Latency: 11.9768
Max latency: 52.6277
Min latency: 11.4904
----Original Message-----
From: Mark Nelson [mailto:mark.nelson@inktank.com]
Sent: Thursday, 20 December 2012 11:59 PM
To: Matthew Anderson
Cc: ceph-devel@vger.kernel.org
Subject: Re: OSD's slow down to a crawl
Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
Mark
On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> Hi Mark,
>
> Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
>
> It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
>
> The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> # ceph --admin-daemon ./ceph-osd.24.asok perf dump
>
> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journ
> al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_bytes
> ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"su
> m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":13
> 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":20
> 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":17102325
> 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
> :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":1708
> .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000},
> "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"avgc
> ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
> :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock":{"w
> ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_lock
> ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
> apply_lo
ck":{"wai
t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_wip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op_latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out_bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000},"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcount":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"subop_push_
in_bytes"
:0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"push":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"numpg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_from_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epoch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":{"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put_sum":7955
46573,"wa
it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":199092,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"put_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
>
> # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
>
> { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> "received_at": "2012-12-20 23:34:42.012131",
> "age": "75.428051",
> "duration": "0.009988",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101498},
> "events": [
> { "time": "2012-12-20 23:34:42.012317",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:42.012425",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:42.012678",
> "event": "started"},
> { "time": "2012-12-20 23:34:42.012720",
> "event": "started"},
> { "time": "2012-12-20 23:34:42.012815",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:34:42.012958",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:34:42.013033",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:34:42.013745",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:34:42.013803",
> "event": "op_commit"},
> { "time": "2012-12-20 23:34:42.017414",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:34:42.022066",
> "event": "op_applied"},
> { "time": "2012-12-20 23:34:42.022119",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:34:42.017048",
> "age": "75.423134",
> "duration": "0.000411",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:34:42.017218",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:42.017363",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:42.017401",
> "event": "started"},
> { "time": "2012-12-20 23:34:42.017459",
> "event": "done"}]},
> { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> "received_at": "2012-12-20 23:34:49.851021",
> "age": "67.589161",
> "duration": "0.007114",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101502},
> "events": [
> { "time": "2012-12-20 23:34:49.851276",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:49.851429",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:49.851616",
> "event": "started"},
> { "time": "2012-12-20 23:34:49.851648",
> "event": "started"},
> { "time": "2012-12-20 23:34:49.851712",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:34:49.851770",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:34:49.851915",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:34:49.853956",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:34:49.853992",
> "event": "op_commit"},
> { "time": "2012-12-20 23:34:49.857016",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:34:49.858077",
> "event": "op_applied"},
> { "time": "2012-12-20 23:34:49.858135",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:34:49.856762",
> "age": "67.583420",
> "duration": "0.000308",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:34:49.856869",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:34:49.856969",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:34:49.857001",
> "event": "started"},
> { "time": "2012-12-20 23:34:49.857070",
> "event": "done"}]},
> { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> "received_at": "2012-12-20 23:35:34.351518",
> "age": "23.088664",
> "duration": "0.009430",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101504},
> "events": [
> { "time": "2012-12-20 23:35:34.351714",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:34.351811",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:34.352030",
> "event": "started"},
> { "time": "2012-12-20 23:35:34.352071",
> "event": "started"},
> { "time": "2012-12-20 23:35:34.352146",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:35:34.352213",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:35:34.352287",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:35:34.355390",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:35:34.355475",
> "event": "op_commit"},
> { "time": "2012-12-20 23:35:34.356661",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:35:34.360894",
> "event": "op_applied"},
> { "time": "2012-12-20 23:35:34.360948",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:35:34.356286",
> "age": "23.083896",
> "duration": "0.000421",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:35:34.356489",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:34.356598",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:34.356640",
> "event": "started"},
> { "time": "2012-12-20 23:35:34.356707",
> "event": "done"}]},
> { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> "received_at": "2012-12-20 23:35:42.133103",
> "age": "15.307079",
> "duration": "0.008357",
> "flag_point": "waiting for sub ops",
> "client_info": { "client": "client.7636",
> "tid": 101512},
> "events": [
> { "time": "2012-12-20 23:35:42.133315",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:42.133427",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:42.133684",
> "event": "started"},
> { "time": "2012-12-20 23:35:42.133723",
> "event": "started"},
> { "time": "2012-12-20 23:35:42.133796",
> "event": "sub_op_sent"},
> { "time": "2012-12-20 23:35:42.133908",
> "event": "commit_queued_for_journal_write"},
> { "time": "2012-12-20 23:35:42.133995",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-12-20 23:35:42.136965",
> "event": "journaled_completion_queued"},
> { "time": "2012-12-20 23:35:42.137025",
> "event": "op_commit"},
> { "time": "2012-12-20 23:35:42.138456",
> "event": "sub_op_commit_rec"},
> { "time": "2012-12-20 23:35:42.141408",
> "event": "op_applied"},
> { "time": "2012-12-20 23:35:42.141460",
> "event": "done"}]},
> { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> "received_at": "2012-12-20 23:35:42.138087",
> "age": "15.302095",
> "duration": "0.000410",
> "flag_point": "started",
> "events": [
> { "time": "2012-12-20 23:35:42.138248",
> "event": "waiting_for_osdmap"},
> { "time": "2012-12-20 23:35:42.138363",
> "event": "reached_pg"},
> { "time": "2012-12-20 23:35:42.138443",
> "event": "started"},
> { "time": "2012-12-20 23:35:42.138497",
> "event": "done"}]}]}
>
> # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
> "osd.24",
> "cluster": "ceph",
> "none": "0\/5",
> "lockdep": "0\/1",
> "context": "0\/1",
> "crush": "1\/1",
> "mds": "1\/5",
> "mds_balancer": "1\/5",
> "mds_locker": "1\/5",
> "mds_log": "1\/5",
> "mds_log_expire": "1\/5",
> "mds_migrator": "1\/5",
> "buffer": "0\/1",
> "timer": "0\/1",
> "filer": "0\/1",
> "striper": "0\/1",
> "objecter": "0\/1",
> "rados": "0\/5",
> "rbd": "0\/5",
> "journaler": "0\/5",
> "objectcacher": "0\/5",
> "client": "0\/5",
> "osd": "0\/5",
> "optracker": "0\/5",
> "objclass": "0\/5",
> "filestore": "1\/3",
> "journal": "1\/3",
> "ms": "0\/5",
> "mon": "1\/5",
> "monc": "0\/10",
> "paxos": "0\/5",
> "tp": "0\/5",
> "auth": "1\/5",
> "crypto": "1\/5",
> "finisher": "1\/1",
> "heartbeatmap": "1\/5",
> "perfcounter": "1\/5",
> "rgw": "1\/5",
> "hadoop": "1\/5",
> "javaclient": "1\/5",
> "asok": "1\/5",
> "throttle": "1\/1",
> "host": "KVM04",
> "fsid": "00000000-0000-0000-0000-000000000000",
> "public_addr": ":\/0",
> "cluster_addr": ":\/0",
> "public_network": "",
> "cluster_network": "",
> "num_client": "1",
> "monmap": "",
> "mon_host": "",
> "lockdep": "false",
> "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> "daemonize": "true",
> "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> "chdir": "\/",
> "max_open_files": "0",
> "fatal_signal_handlers": "true",
> "log_file": "\/var\/log\/ceph\/osd.24.log",
> "log_max_new": "1000",
> "log_max_recent": "100000",
> "log_to_stderr": "false",
> "err_to_stderr": "true",
> "log_to_syslog": "false",
> "err_to_syslog": "false",
> "log_flush_on_exit": "true",
> "clog_to_monitors": "true",
> "clog_to_syslog": "false",
> "mon_cluster_log_to_syslog": "false",
> "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> "key": "",
> "keyfile": "",
> "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> "heartbeat_interval": "5",
> "heartbeat_file": "",
> "perf": "true",
> "ms_tcp_nodelay": "true",
> "ms_initial_backoff": "0.2",
> "ms_max_backoff": "15",
> "ms_nocrc": "false",
> "ms_die_on_bad_msg": "false",
> "ms_dispatch_throttle_bytes": "104857600",
> "ms_bind_ipv6": "false",
> "ms_bind_port_min": "6800",
> "ms_bind_port_max": "7100",
> "ms_rwthread_stack_bytes": "1048576",
> "ms_tcp_read_timeout": "900",
> "ms_inject_socket_failures": "0",
> "ms_inject_delay_type": "",
> "ms_inject_delay_max": "1",
> "ms_inject_delay_probability": "0",
> "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> "mon_initial_members": "",
> "mon_sync_fs_threshold": "5",
> "mon_tick_interval": "5",
> "mon_subscribe_interval": "300",
> "mon_osd_laggy_halflife": "3600",
> "mon_osd_laggy_weight": "0.3",
> "mon_osd_adjust_heartbeat_grace": "true",
> "mon_osd_adjust_down_out_interval": "true",
> "mon_osd_auto_mark_in": "false",
> "mon_osd_auto_mark_auto_out_in": "true",
> "mon_osd_auto_mark_new_in": "true",
> "mon_osd_down_out_interval": "300",
> "mon_osd_min_up_ratio": "0.3",
> "mon_osd_min_in_ratio": "0.3",
> "mon_lease": "5",
> "mon_lease_renew_interval": "3",
> "mon_lease_ack_timeout": "10",
> "mon_clock_drift_allowed": "0.05",
> "mon_clock_drift_warn_backoff": "5",
> "mon_accept_timeout": "10",
> "mon_pg_create_interval": "30",
> "mon_pg_stuck_threshold": "300",
> "mon_osd_full_ratio": "0.95",
> "mon_osd_nearfull_ratio": "0.85",
> "mon_globalid_prealloc": "100",
> "mon_osd_report_timeout": "900",
> "mon_force_standby_active": "true",
> "mon_min_osdmap_epochs": "500",
> "mon_max_pgmap_epochs": "500",
> "mon_max_log_epochs": "500",
> "mon_max_osd": "10000",
> "mon_probe_timeout": "2",
> "mon_slurp_timeout": "10",
> "mon_slurp_bytes": "262144",
> "mon_client_bytes": "104857600",
> "mon_daemon_bytes": "419430400",
> "mon_max_log_entries_per_event": "4096",
> "paxos_max_join_drift": "10",
> "paxos_propose_interval": "1",
> "paxos_min_wait": "0.05",
> "clock_offset": "0",
> "auth_cluster_required": "none",
> "auth_service_required": "none",
> "auth_client_required": "none",
> "auth_supported": "none",
> "cephx_require_signatures": "false",
> "cephx_sign_messages": "true",
> "auth_mon_ticket_ttl": "43200",
> "auth_service_ticket_ttl": "3600",
> "mon_client_hunt_interval": "3",
> "mon_client_ping_interval": "10",
> "mon_client_max_log_entries_per_message": "1000",
> "mon_max_pool_pg_num": "65536",
> "client_cache_size": "16384",
> "client_cache_mid": "0.75",
> "client_use_random_mds": "false",
> "client_mount_timeout": "30",
> "client_tick_interval": "1",
> "client_trace": "",
> "client_readahead_min": "131072",
> "client_readahead_max_bytes": "0",
> "client_readahead_max_periods": "4",
> "client_snapdir": ".snap",
> "client_mountpoint": "\/",
> "client_notify_timeout": "10",
> "client_caps_release_delay": "5",
> "client_oc": "true",
> "client_oc_size": "209715200",
> "client_oc_max_dirty": "104857600",
> "client_oc_target_dirty": "8388608",
> "client_oc_max_dirty_age": "5",
> "client_oc_max_objects": "1000",
> "client_debug_force_sync_read": "false",
> "client_debug_inject_tick_delay": "0",
> "fuse_use_invalidate_cb": "false",
> "fuse_big_writes": "true",
> "fuse_debug": "false",
> "objecter_tick_interval": "5",
> "objecter_timeout": "10",
> "objecter_inflight_op_bytes": "104857600",
> "objecter_inflight_ops": "1024",
> "journaler_allow_split_entries": "true",
> "journaler_write_head_interval": "15",
> "journaler_prefetch_periods": "10",
> "journaler_prezero_periods": "5",
> "journaler_batch_interval": "0.001",
> "journaler_batch_max": "0",
> "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> "mds_max_file_size": "1099511627776",
> "mds_cache_size": "100000",
> "mds_cache_mid": "0.7",
> "mds_mem_max": "1048576",
> "mds_dir_commit_ratio": "0.5",
> "mds_dir_max_commit_size": "90",
> "mds_decay_halflife": "5",
> "mds_beacon_interval": "4",
> "mds_beacon_grace": "15",
> "mds_blacklist_interval": "1440",
> "mds_session_timeout": "60",
> "mds_session_autoclose": "300",
> "mds_reconnect_timeout": "45",
> "mds_tick_interval": "5",
> "mds_dirstat_min_interval": "1",
> "mds_scatter_nudge_interval": "5",
> "mds_client_prealloc_inos": "1000",
> "mds_early_reply": "true",
> "mds_use_tmap": "true",
> "mds_default_dir_hash": "2",
> "mds_log": "true",
> "mds_log_skip_corrupt_events": "false",
> "mds_log_max_events": "-1",
> "mds_log_max_segments": "30",
> "mds_log_max_expiring": "20",
> "mds_bal_sample_interval": "3",
> "mds_bal_replicate_threshold": "8000",
> "mds_bal_unreplicate_threshold": "0",
> "mds_bal_frag": "false",
> "mds_bal_split_size": "10000",
> "mds_bal_split_rd": "25000",
> "mds_bal_split_wr": "10000",
> "mds_bal_split_bits": "3",
> "mds_bal_merge_size": "50",
> "mds_bal_merge_rd": "1000",
> "mds_bal_merge_wr": "1000",
> "mds_bal_interval": "10",
> "mds_bal_fragment_interval": "5",
> "mds_bal_idle_threshold": "0",
> "mds_bal_max": "-1",
> "mds_bal_max_until": "-1",
> "mds_bal_mode": "0",
> "mds_bal_min_rebalance": "0.1",
> "mds_bal_min_start": "0.2",
> "mds_bal_need_min": "0.8",
> "mds_bal_need_max": "1.2",
> "mds_bal_midchunk": "0.3",
> "mds_bal_minchunk": "0.001",
> "mds_bal_target_removal_min": "5",
> "mds_bal_target_removal_max": "10",
> "mds_replay_interval": "1",
> "mds_shutdown_check": "0",
> "mds_thrash_exports": "0",
> "mds_thrash_fragments": "0",
> "mds_dump_cache_on_map": "false",
> "mds_dump_cache_after_rejoin": "false",
> "mds_verify_scatter": "false",
> "mds_debug_scatterstat": "false",
> "mds_debug_frag": "false",
> "mds_debug_auth_pins": "false",
> "mds_debug_subtrees": "false",
> "mds_kill_mdstable_at": "0",
> "mds_kill_export_at": "0",
> "mds_kill_import_at": "0",
> "mds_kill_link_at": "0",
> "mds_kill_rename_at": "0",
> "mds_wipe_sessions": "false",
> "mds_wipe_ino_prealloc": "false",
> "mds_skip_ino": "0",
> "max_mds": "1",
> "mds_standby_for_name": "",
> "mds_standby_for_rank": "-1",
> "mds_standby_replay": "false",
> "osd_auto_upgrade_tmap": "true",
> "osd_tmapput_sets_uses_tmap": "false",
> "osd_max_backfills": "10",
> "osd_backfill_full_ratio": "0.85",
> "osd_backfill_retry_interval": "10",
> "osd_uuid": "00000000-0000-0000-0000-000000000000",
> "osd_data": "\/srv\/ceph\/osd\/osd.24",
> "osd_journal": "\/dev\/sda3",
> "osd_journal_size": "5120",
> "osd_max_write_size": "90",
> "osd_max_pgls": "1024",
> "osd_client_message_size_cap": "524288000",
> "osd_pg_bits": "6",
> "osd_pgp_bits": "6",
> "osd_min_rep": "1",
> "osd_max_rep": "10",
> "osd_pool_default_crush_rule": "0",
> "osd_pool_default_size": "2",
> "osd_pool_default_min_size": "0",
> "osd_pool_default_pg_num": "8",
> "osd_pool_default_pgp_num": "8",
> "osd_map_dedup": "true",
> "osd_map_cache_size": "500",
> "osd_map_message_max": "100",
> "osd_op_threads": "2",
> "osd_disk_threads": "1",
> "osd_recovery_threads": "1",
> "osd_recover_clone_overlap": "true",
> "osd_backfill_scan_min": "64",
> "osd_backfill_scan_max": "512",
> "osd_op_thread_timeout": "30",
> "osd_recovery_thread_timeout": "30",
> "osd_snap_trim_thread_timeout": "3600",
> "osd_scrub_thread_timeout": "60",
> "osd_scrub_finalize_thread_timeout": "600",
> "osd_remove_thread_timeout": "3600",
> "osd_command_thread_timeout": "600",
> "osd_age": "0.8",
> "osd_age_time": "0",
> "osd_heartbeat_addr": ":\/0",
> "osd_heartbeat_interval": "6",
> "osd_heartbeat_grace": "20",
> "osd_mon_heartbeat_interval": "30",
> "osd_mon_report_interval_max": "120",
> "osd_mon_report_interval_min": "5",
> "osd_mon_ack_timeout": "30",
> "osd_min_down_reporters": "1",
> "osd_min_down_reports": "3",
> "osd_default_data_pool_replay_window": "45",
> "osd_preserve_trimmed_log": "false",
> "osd_auto_mark_unfound_lost": "false",
> "osd_recovery_delay_start": "0",
> "osd_recovery_max_active": "50",
> "osd_recovery_max_chunk": "8388608",
> "osd_recovery_forget_lost_objects": "false",
> "osd_max_scrubs": "1",
> "osd_scrub_load_threshold": "0.5",
> "osd_scrub_min_interval": "300",
> "osd_scrub_max_interval": "86400",
> "osd_deep_scrub_interval": "604800",
> "osd_deep_scrub_stride": "524288",
> "osd_auto_weight": "false",
> "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> "osd_check_for_log_corruption": "false",
> "osd_use_stale_snap": "false",
> "osd_rollback_to_cluster_snap": "",
> "osd_default_notify_timeout": "30",
> "osd_kill_backfill_at": "0",
> "osd_min_pg_log_entries": "1000",
> "osd_op_complaint_time": "30",
> "osd_command_max_records": "256",
> "osd_op_log_threshold": "5",
> "osd_verify_sparse_read_holes": "false",
> "osd_debug_drop_ping_probability": "0",
> "osd_debug_drop_ping_duration": "0",
> "osd_debug_drop_pg_create_probability": "0",
> "osd_debug_drop_pg_create_duration": "1",
> "osd_debug_drop_op_probability": "0",
> "osd_op_history_size": "20",
> "osd_op_history_duration": "600",
> "osd_target_transaction_size": "300",
> "osd_client_op_priority": "63",
> "osd_recovery_op_priority": "10",
> "filestore": "false",
> "filestore_index_retry_probability": "0",
> "filestore_debug_omap_check": "false",
> "filestore_xattr_use_omap": "false",
> "filestore_max_inline_xattr_size": "512",
> "filestore_max_inline_xattrs": "2",
> "filestore_max_sync_interval": "5",
> "filestore_min_sync_interval": "0.01",
> "filestore_btrfs_snap": "true",
> "filestore_btrfs_clone_range": "true",
> "filestore_fsync_flushes_journal_data": "false",
> "filestore_fiemap": "false",
> "filestore_flusher": "true",
> "filestore_flusher_max_fds": "512",
> "filestore_flush_min": "65536",
> "filestore_sync_flush": "false",
> "filestore_journal_parallel": "false",
> "filestore_journal_writeahead": "false",
> "filestore_journal_trailing": "false",
> "filestore_queue_max_ops": "500",
> "filestore_queue_max_bytes": "104857600",
> "filestore_queue_committing_max_ops": "500",
> "filestore_queue_committing_max_bytes": "104857600",
> "filestore_op_threads": "2",
> "filestore_op_thread_timeout": "60",
> "filestore_op_thread_suicide_timeout": "180",
> "filestore_commit_timeout": "600",
> "filestore_fiemap_threshold": "4096",
> "filestore_merge_threshold": "10",
> "filestore_split_multiple": "2",
> "filestore_update_to": "1000",
> "filestore_blackhole": "false",
> "filestore_dump_file": "",
> "filestore_kill_at": "0",
> "filestore_fail_eio": "true",
> "journal_dio": "true",
> "journal_aio": "false",
> "journal_block_align": "true",
> "journal_max_write_bytes": "10485760",
> "journal_max_write_entries": "100",
> "journal_queue_max_ops": "500",
> "journal_queue_max_bytes": "104857600",
> "journal_align_min_size": "65536",
> "journal_replay_from": "0",
> "journal_zero_on_create": "false",
> "rbd_cache": "false",
> "rbd_cache_size": "33554432",
> "rbd_cache_max_dirty": "25165824",
> "rbd_cache_target_dirty": "16777216",
> "rbd_cache_max_dirty_age": "1",
> "nss_db_path": "",
> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> "rgw_enable_apis": "s3, swift, swift_auth, admin",
> "rgw_cache_enabled": "true",
> "rgw_cache_lru_size": "10000",
> "rgw_socket_path": "",
> "rgw_dns_name": "",
> "rgw_swift_url": "",
> "rgw_swift_url_prefix": "swift",
> "rgw_swift_auth_url": "",
> "rgw_swift_auth_entry": "auth",
> "rgw_keystone_url": "",
> "rgw_keystone_admin_token": "",
> "rgw_keystone_accepted_roles": "Member, admin",
> "rgw_keystone_token_cache_size": "10000",
> "rgw_keystone_revocation_interval": "900",
> "rgw_admin_entry": "admin",
> "rgw_enforce_swift_acls": "true",
> "rgw_swift_token_expiration": "86400",
> "rgw_print_continue": "true",
> "rgw_remote_addr_param": "REMOTE_ADDR",
> "rgw_op_thread_timeout": "600",
> "rgw_op_thread_suicide_timeout": "0",
> "rgw_thread_pool_size": "100",
> "rgw_num_control_oids": "8",
> "rgw_cluster_root_pool": ".rgw.root",
> "rgw_log_nonexistent_bucket": "false",
> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> "rgw_log_object_name_utc": "false",
> "rgw_usage_max_shards": "32",
> "rgw_usage_max_user_shards": "1",
> "rgw_enable_ops_log": "true",
> "rgw_enable_usage_log": "true",
> "rgw_ops_log_rados": "true",
> "rgw_ops_log_socket_path": "",
> "rgw_ops_log_data_backlog": "5242880",
> "rgw_usage_log_flush_threshold": "1024",
> "rgw_usage_log_tick_interval": "30",
> "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> "rgw_intent_log_object_name_utc": "false",
> "rgw_init_timeout": "30",
> "rgw_mime_types_file": "\/etc\/mime.types",
> "rgw_gc_max_objs": "32",
> "rgw_gc_obj_min_wait": "7200",
> "rgw_gc_processor_max_time": "3600",
> "rgw_gc_processor_period": "3600",
> "rgw_s3_success_create_obj_status": "0",
> "rgw_resolve_cname": "false",
> "rgw_obj_stripe_size": "4194304",
> "rgw_extended_http_attrs": "",
> "mutex_perf_counter": "false",
> "internal_safe_to_start_threads": "true"}
>
>
> -----Original Message-----
> From: Mark Nelson [mailto:mark.nelson@inktank.com]
> Sent: Thursday, 20 December 2012 11:28 PM
> To: Matthew Anderson
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: OSD's slow down to a crawl
>
> Hi Matt,
>
> Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> Sebastien Han made a nice little blog entry describing it:
>
> http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
>
> You can get a lot of information about the current state of things.
>
> Mark
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: OSD's slow down to a crawl
2012-12-20 16:08 ` Matthew Anderson
@ 2012-12-20 16:22 ` Mark Nelson
2012-12-20 16:34 ` Matthew Anderson
2012-12-20 16:30 ` Sage Weil
1 sibling, 1 reply; 13+ messages in thread
From: Mark Nelson @ 2012-12-20 16:22 UTC (permalink / raw)
To: Matthew Anderson; +Cc: ceph-devel@vger.kernel.org
Ok,
So the interesting news here is that the OSDs don't appear to be stalled
exactly, but they do seem to be going really slow. With that new of a
kernel and 0.55 you should have syncfs support so I don't think that's
the problem. What kind of networking? Any possible problems there?
Mark
On 12/20/2012 10:08 AM, Matthew Anderson wrote:
> # rados bench 60 write -t 256 -p ssd
> Maintaining 256 concurrent writes of 4194304 bytes for at least 60 seconds.
> Object prefix: benchmark_data_KVM03_12985
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 45 45 0 0 0 - 0
> 2 71 71 0 0 0 - 0
> 3 90 90 0 0 0 - 0
> 4 125 125 0 0 0 - 0
> 5 136 136 0 0 0 - 0
> 6 144 144 0 0 0 - 0
> 7 175 175 0 0 0 - 0
> 8 201 201 0 0 0 - 0
> 9 217 217 0 0 0 - 0
> 10 228 228 0 0 0 - 0
> 11 251 251 0 0 0 - 0
> 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> 13 255 271 16 4.92194 20 12.9795 11.9979
> 14 255 291 36 10.2833 80 13.1795 12.6478
> 15 255 301 46 12.2639 40 13.9424 12.9123
> 16 255 303 48 11.9973 8 14.0277 12.9579
> 17 255 321 66 15.526 72 15.1408 13.4616
> 18 255 345 90 19.9956 96 15.5673 13.9937
> 19 255 349 94 19.7851 16 15.707 14.0635
> 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 20 255 361 106 21.1953 48 16.4053 14.3121
> 21 255 367 112 21.3285 24 17.4263 14.4795
> 22 255 370 115 20.9043 12 17.3579 14.5544
> 23 255 387 132 22.9513 68 18.3185 15.0867
> 24 255 394 139 23.1614 28 18.2868 15.2504
> 25 255 404 149 23.8346 40 18.741 15.5048
> 26 255 410 155 23.8407 24 19.032 15.6358
> 27 255 410 155 22.9577 0 - 15.6358
> 28 255 410 155 22.1378 0 - 15.6358
> 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> 30 255 431 176 23.4614 80 22.6577 16.4822
> 31 255 447 192 24.7687 64 23.2345 17.0563
> 32 255 464 209 26.1192 68 23.3788 17.5859
> 33 255 485 230 27.8726 84 22.8924 18.1122
> 34 255 497 242 28.4642 48 23.563 18.3693
> 35 255 508 253 28.9078 44 23.0516 18.5935
> 36 255 514 259 28.7713 24 24.4415 18.7278
> 37 255 530 275 29.723 64 23.847 19.0424
> 38 255 541 286 30.0985 44 24.5757 19.2427
> 39 255 544 289 29.6344 12 24.6802 19.2984
> 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 40 255 544 289 28.8935 0 - 19.2984
> 41 255 544 289 28.1888 0 - 19.2984
> 42 255 544 289 27.5176 0 - 19.2984
> 43 255 544 289 26.8776 0 - 19.2984
> 44 255 544 289 26.2668 0 - 19.2984
> 45 255 544 289 25.6831 0 - 19.2984
> 46 255 544 289 25.1248 0 - 19.2984
> 47 255 544 289 24.5902 0 - 19.2984
> 48 255 544 289 24.0779 0 - 19.2984
> 49 255 544 289 23.5865 0 - 19.2984
> 50 255 544 289 23.1148 0 - 19.2984
> 51 255 544 289 22.6616 0 - 19.2984
> 52 255 544 289 22.2258 0 - 19.2984
> 53 255 544 289 21.8065 0 - 19.2984
> 54 255 544 289 21.4026 0 - 19.2984
> 55 255 544 289 21.0135 0 - 19.2984
> 56 255 544 289 20.6383 0 - 19.2984
> 57 255 544 289 20.2762 0 - 19.2984
> 58 255 544 289 19.9266 0 - 19.2984
> 59 255 544 289 19.5889 0 - 19.2984
> 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 60 255 544 289 19.2624 0 - 19.2984
> 61 255 544 289 18.9466 0 - 19.2984
> 62 255 544 289 18.641 0 - 19.2984
> 63 255 544 289 18.3452 0 - 19.2984
> 64 255 544 289 18.0585 0 - 19.2984
> 65 255 544 289 17.7807 0 - 19.2984
> 66 255 544 289 17.5113 0 - 19.2984
> 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> Total time run: 68.000740
> Total writes made: 545
> Write size: 4194304
> Bandwidth (MB/sec): 32.058
>
> Stddev Bandwidth: 26.8837
> Max bandwidth (MB/sec): 96
> Min bandwidth (MB/sec): 0
> Average Latency: 29.2801
> Stddev Latency: 11.9768
> Max latency: 52.6277
> Min latency: 11.4904
>
>
>
>
>
> ----Original Message-----
> From: Mark Nelson [mailto:mark.nelson@inktank.com]
> Sent: Thursday, 20 December 2012 11:59 PM
> To: Matthew Anderson
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: OSD's slow down to a crawl
>
> Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
>
> Mark
>
> On 12/20/2012 09:51 AM, Matthew Anderson wrote:
>> Hi Mark,
>>
>> Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
>>
>> It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
>>
>> The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
>> # ceph --admin-daemon ./ceph-osd.24.asok perf dump
>>
>> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journ
>> al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_bytes
>> ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"su
>> m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":13
>> 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":20
>> 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":17102325
>> 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
>> :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":1708
>> .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000},
>> "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"avgc
>> ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
>> :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock":{"w
>> ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_lock
>> ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
>> apply_lo
> ck":{"wai
> t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_wip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op_latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out_bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000},"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcount":5700,"sum":26.877085000},
"subop_w":0,"subop_w_in_bytes":813224156,"subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"subop_push_
> in_bytes"
> :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"push":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"numpg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_from_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epoch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgcount":0,"sum":0.000000000}},
"throttle-msgr_dispatch_throttler-client":{"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put_sum":7955
> 46573,"wa
> it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":199092,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes":{"val":0,"max":524288000,"
get":31308,"get_sum":795353720,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"put_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
>>
>> # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
>>
>> { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
>> "received_at": "2012-12-20 23:34:42.012131",
>> "age": "75.428051",
>> "duration": "0.009988",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101498},
>> "events": [
>> { "time": "2012-12-20 23:34:42.012317",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:42.012425",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:42.012678",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:42.012720",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:42.012815",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:34:42.012958",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:34:42.013033",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:34:42.013745",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:34:42.013803",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:34:42.017414",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:34:42.022066",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:34:42.022119",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:34:42.017048",
>> "age": "75.423134",
>> "duration": "0.000411",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:34:42.017218",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:42.017363",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:42.017401",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:42.017459",
>> "event": "done"}]},
>> { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
>> "received_at": "2012-12-20 23:34:49.851021",
>> "age": "67.589161",
>> "duration": "0.007114",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101502},
>> "events": [
>> { "time": "2012-12-20 23:34:49.851276",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:49.851429",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:49.851616",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:49.851648",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:49.851712",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:34:49.851770",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:34:49.851915",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:34:49.853956",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:34:49.853992",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:34:49.857016",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:34:49.858077",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:34:49.858135",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:34:49.856762",
>> "age": "67.583420",
>> "duration": "0.000308",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:34:49.856869",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:49.856969",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:49.857001",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:49.857070",
>> "event": "done"}]},
>> { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
>> "received_at": "2012-12-20 23:35:34.351518",
>> "age": "23.088664",
>> "duration": "0.009430",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101504},
>> "events": [
>> { "time": "2012-12-20 23:35:34.351714",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:34.351811",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:34.352030",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:34.352071",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:34.352146",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:35:34.352213",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:35:34.352287",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:35:34.355390",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:35:34.355475",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:35:34.356661",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:35:34.360894",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:35:34.360948",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:35:34.356286",
>> "age": "23.083896",
>> "duration": "0.000421",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:35:34.356489",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:34.356598",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:34.356640",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:34.356707",
>> "event": "done"}]},
>> { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
>> "received_at": "2012-12-20 23:35:42.133103",
>> "age": "15.307079",
>> "duration": "0.008357",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101512},
>> "events": [
>> { "time": "2012-12-20 23:35:42.133315",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:42.133427",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:42.133684",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:42.133723",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:42.133796",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:35:42.133908",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:35:42.133995",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:35:42.136965",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:35:42.137025",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:35:42.138456",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:35:42.141408",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:35:42.141460",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:35:42.138087",
>> "age": "15.302095",
>> "duration": "0.000410",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:35:42.138248",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:42.138363",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:42.138443",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:42.138497",
>> "event": "done"}]}]}
>>
>> # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
>> "osd.24",
>> "cluster": "ceph",
>> "none": "0\/5",
>> "lockdep": "0\/1",
>> "context": "0\/1",
>> "crush": "1\/1",
>> "mds": "1\/5",
>> "mds_balancer": "1\/5",
>> "mds_locker": "1\/5",
>> "mds_log": "1\/5",
>> "mds_log_expire": "1\/5",
>> "mds_migrator": "1\/5",
>> "buffer": "0\/1",
>> "timer": "0\/1",
>> "filer": "0\/1",
>> "striper": "0\/1",
>> "objecter": "0\/1",
>> "rados": "0\/5",
>> "rbd": "0\/5",
>> "journaler": "0\/5",
>> "objectcacher": "0\/5",
>> "client": "0\/5",
>> "osd": "0\/5",
>> "optracker": "0\/5",
>> "objclass": "0\/5",
>> "filestore": "1\/3",
>> "journal": "1\/3",
>> "ms": "0\/5",
>> "mon": "1\/5",
>> "monc": "0\/10",
>> "paxos": "0\/5",
>> "tp": "0\/5",
>> "auth": "1\/5",
>> "crypto": "1\/5",
>> "finisher": "1\/1",
>> "heartbeatmap": "1\/5",
>> "perfcounter": "1\/5",
>> "rgw": "1\/5",
>> "hadoop": "1\/5",
>> "javaclient": "1\/5",
>> "asok": "1\/5",
>> "throttle": "1\/1",
>> "host": "KVM04",
>> "fsid": "00000000-0000-0000-0000-000000000000",
>> "public_addr": ":\/0",
>> "cluster_addr": ":\/0",
>> "public_network": "",
>> "cluster_network": "",
>> "num_client": "1",
>> "monmap": "",
>> "mon_host": "",
>> "lockdep": "false",
>> "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
>> "daemonize": "true",
>> "pid_file": "\/var\/run\/ceph\/osd.24.pid",
>> "chdir": "\/",
>> "max_open_files": "0",
>> "fatal_signal_handlers": "true",
>> "log_file": "\/var\/log\/ceph\/osd.24.log",
>> "log_max_new": "1000",
>> "log_max_recent": "100000",
>> "log_to_stderr": "false",
>> "err_to_stderr": "true",
>> "log_to_syslog": "false",
>> "err_to_syslog": "false",
>> "log_flush_on_exit": "true",
>> "clog_to_monitors": "true",
>> "clog_to_syslog": "false",
>> "mon_cluster_log_to_syslog": "false",
>> "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
>> "key": "",
>> "keyfile": "",
>> "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
>> "heartbeat_interval": "5",
>> "heartbeat_file": "",
>> "perf": "true",
>> "ms_tcp_nodelay": "true",
>> "ms_initial_backoff": "0.2",
>> "ms_max_backoff": "15",
>> "ms_nocrc": "false",
>> "ms_die_on_bad_msg": "false",
>> "ms_dispatch_throttle_bytes": "104857600",
>> "ms_bind_ipv6": "false",
>> "ms_bind_port_min": "6800",
>> "ms_bind_port_max": "7100",
>> "ms_rwthread_stack_bytes": "1048576",
>> "ms_tcp_read_timeout": "900",
>> "ms_inject_socket_failures": "0",
>> "ms_inject_delay_type": "",
>> "ms_inject_delay_max": "1",
>> "ms_inject_delay_probability": "0",
>> "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
>> "mon_initial_members": "",
>> "mon_sync_fs_threshold": "5",
>> "mon_tick_interval": "5",
>> "mon_subscribe_interval": "300",
>> "mon_osd_laggy_halflife": "3600",
>> "mon_osd_laggy_weight": "0.3",
>> "mon_osd_adjust_heartbeat_grace": "true",
>> "mon_osd_adjust_down_out_interval": "true",
>> "mon_osd_auto_mark_in": "false",
>> "mon_osd_auto_mark_auto_out_in": "true",
>> "mon_osd_auto_mark_new_in": "true",
>> "mon_osd_down_out_interval": "300",
>> "mon_osd_min_up_ratio": "0.3",
>> "mon_osd_min_in_ratio": "0.3",
>> "mon_lease": "5",
>> "mon_lease_renew_interval": "3",
>> "mon_lease_ack_timeout": "10",
>> "mon_clock_drift_allowed": "0.05",
>> "mon_clock_drift_warn_backoff": "5",
>> "mon_accept_timeout": "10",
>> "mon_pg_create_interval": "30",
>> "mon_pg_stuck_threshold": "300",
>> "mon_osd_full_ratio": "0.95",
>> "mon_osd_nearfull_ratio": "0.85",
>> "mon_globalid_prealloc": "100",
>> "mon_osd_report_timeout": "900",
>> "mon_force_standby_active": "true",
>> "mon_min_osdmap_epochs": "500",
>> "mon_max_pgmap_epochs": "500",
>> "mon_max_log_epochs": "500",
>> "mon_max_osd": "10000",
>> "mon_probe_timeout": "2",
>> "mon_slurp_timeout": "10",
>> "mon_slurp_bytes": "262144",
>> "mon_client_bytes": "104857600",
>> "mon_daemon_bytes": "419430400",
>> "mon_max_log_entries_per_event": "4096",
>> "paxos_max_join_drift": "10",
>> "paxos_propose_interval": "1",
>> "paxos_min_wait": "0.05",
>> "clock_offset": "0",
>> "auth_cluster_required": "none",
>> "auth_service_required": "none",
>> "auth_client_required": "none",
>> "auth_supported": "none",
>> "cephx_require_signatures": "false",
>> "cephx_sign_messages": "true",
>> "auth_mon_ticket_ttl": "43200",
>> "auth_service_ticket_ttl": "3600",
>> "mon_client_hunt_interval": "3",
>> "mon_client_ping_interval": "10",
>> "mon_client_max_log_entries_per_message": "1000",
>> "mon_max_pool_pg_num": "65536",
>> "client_cache_size": "16384",
>> "client_cache_mid": "0.75",
>> "client_use_random_mds": "false",
>> "client_mount_timeout": "30",
>> "client_tick_interval": "1",
>> "client_trace": "",
>> "client_readahead_min": "131072",
>> "client_readahead_max_bytes": "0",
>> "client_readahead_max_periods": "4",
>> "client_snapdir": ".snap",
>> "client_mountpoint": "\/",
>> "client_notify_timeout": "10",
>> "client_caps_release_delay": "5",
>> "client_oc": "true",
>> "client_oc_size": "209715200",
>> "client_oc_max_dirty": "104857600",
>> "client_oc_target_dirty": "8388608",
>> "client_oc_max_dirty_age": "5",
>> "client_oc_max_objects": "1000",
>> "client_debug_force_sync_read": "false",
>> "client_debug_inject_tick_delay": "0",
>> "fuse_use_invalidate_cb": "false",
>> "fuse_big_writes": "true",
>> "fuse_debug": "false",
>> "objecter_tick_interval": "5",
>> "objecter_timeout": "10",
>> "objecter_inflight_op_bytes": "104857600",
>> "objecter_inflight_ops": "1024",
>> "journaler_allow_split_entries": "true",
>> "journaler_write_head_interval": "15",
>> "journaler_prefetch_periods": "10",
>> "journaler_prezero_periods": "5",
>> "journaler_batch_interval": "0.001",
>> "journaler_batch_max": "0",
>> "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
>> "mds_max_file_size": "1099511627776",
>> "mds_cache_size": "100000",
>> "mds_cache_mid": "0.7",
>> "mds_mem_max": "1048576",
>> "mds_dir_commit_ratio": "0.5",
>> "mds_dir_max_commit_size": "90",
>> "mds_decay_halflife": "5",
>> "mds_beacon_interval": "4",
>> "mds_beacon_grace": "15",
>> "mds_blacklist_interval": "1440",
>> "mds_session_timeout": "60",
>> "mds_session_autoclose": "300",
>> "mds_reconnect_timeout": "45",
>> "mds_tick_interval": "5",
>> "mds_dirstat_min_interval": "1",
>> "mds_scatter_nudge_interval": "5",
>> "mds_client_prealloc_inos": "1000",
>> "mds_early_reply": "true",
>> "mds_use_tmap": "true",
>> "mds_default_dir_hash": "2",
>> "mds_log": "true",
>> "mds_log_skip_corrupt_events": "false",
>> "mds_log_max_events": "-1",
>> "mds_log_max_segments": "30",
>> "mds_log_max_expiring": "20",
>> "mds_bal_sample_interval": "3",
>> "mds_bal_replicate_threshold": "8000",
>> "mds_bal_unreplicate_threshold": "0",
>> "mds_bal_frag": "false",
>> "mds_bal_split_size": "10000",
>> "mds_bal_split_rd": "25000",
>> "mds_bal_split_wr": "10000",
>> "mds_bal_split_bits": "3",
>> "mds_bal_merge_size": "50",
>> "mds_bal_merge_rd": "1000",
>> "mds_bal_merge_wr": "1000",
>> "mds_bal_interval": "10",
>> "mds_bal_fragment_interval": "5",
>> "mds_bal_idle_threshold": "0",
>> "mds_bal_max": "-1",
>> "mds_bal_max_until": "-1",
>> "mds_bal_mode": "0",
>> "mds_bal_min_rebalance": "0.1",
>> "mds_bal_min_start": "0.2",
>> "mds_bal_need_min": "0.8",
>> "mds_bal_need_max": "1.2",
>> "mds_bal_midchunk": "0.3",
>> "mds_bal_minchunk": "0.001",
>> "mds_bal_target_removal_min": "5",
>> "mds_bal_target_removal_max": "10",
>> "mds_replay_interval": "1",
>> "mds_shutdown_check": "0",
>> "mds_thrash_exports": "0",
>> "mds_thrash_fragments": "0",
>> "mds_dump_cache_on_map": "false",
>> "mds_dump_cache_after_rejoin": "false",
>> "mds_verify_scatter": "false",
>> "mds_debug_scatterstat": "false",
>> "mds_debug_frag": "false",
>> "mds_debug_auth_pins": "false",
>> "mds_debug_subtrees": "false",
>> "mds_kill_mdstable_at": "0",
>> "mds_kill_export_at": "0",
>> "mds_kill_import_at": "0",
>> "mds_kill_link_at": "0",
>> "mds_kill_rename_at": "0",
>> "mds_wipe_sessions": "false",
>> "mds_wipe_ino_prealloc": "false",
>> "mds_skip_ino": "0",
>> "max_mds": "1",
>> "mds_standby_for_name": "",
>> "mds_standby_for_rank": "-1",
>> "mds_standby_replay": "false",
>> "osd_auto_upgrade_tmap": "true",
>> "osd_tmapput_sets_uses_tmap": "false",
>> "osd_max_backfills": "10",
>> "osd_backfill_full_ratio": "0.85",
>> "osd_backfill_retry_interval": "10",
>> "osd_uuid": "00000000-0000-0000-0000-000000000000",
>> "osd_data": "\/srv\/ceph\/osd\/osd.24",
>> "osd_journal": "\/dev\/sda3",
>> "osd_journal_size": "5120",
>> "osd_max_write_size": "90",
>> "osd_max_pgls": "1024",
>> "osd_client_message_size_cap": "524288000",
>> "osd_pg_bits": "6",
>> "osd_pgp_bits": "6",
>> "osd_min_rep": "1",
>> "osd_max_rep": "10",
>> "osd_pool_default_crush_rule": "0",
>> "osd_pool_default_size": "2",
>> "osd_pool_default_min_size": "0",
>> "osd_pool_default_pg_num": "8",
>> "osd_pool_default_pgp_num": "8",
>> "osd_map_dedup": "true",
>> "osd_map_cache_size": "500",
>> "osd_map_message_max": "100",
>> "osd_op_threads": "2",
>> "osd_disk_threads": "1",
>> "osd_recovery_threads": "1",
>> "osd_recover_clone_overlap": "true",
>> "osd_backfill_scan_min": "64",
>> "osd_backfill_scan_max": "512",
>> "osd_op_thread_timeout": "30",
>> "osd_recovery_thread_timeout": "30",
>> "osd_snap_trim_thread_timeout": "3600",
>> "osd_scrub_thread_timeout": "60",
>> "osd_scrub_finalize_thread_timeout": "600",
>> "osd_remove_thread_timeout": "3600",
>> "osd_command_thread_timeout": "600",
>> "osd_age": "0.8",
>> "osd_age_time": "0",
>> "osd_heartbeat_addr": ":\/0",
>> "osd_heartbeat_interval": "6",
>> "osd_heartbeat_grace": "20",
>> "osd_mon_heartbeat_interval": "30",
>> "osd_mon_report_interval_max": "120",
>> "osd_mon_report_interval_min": "5",
>> "osd_mon_ack_timeout": "30",
>> "osd_min_down_reporters": "1",
>> "osd_min_down_reports": "3",
>> "osd_default_data_pool_replay_window": "45",
>> "osd_preserve_trimmed_log": "false",
>> "osd_auto_mark_unfound_lost": "false",
>> "osd_recovery_delay_start": "0",
>> "osd_recovery_max_active": "50",
>> "osd_recovery_max_chunk": "8388608",
>> "osd_recovery_forget_lost_objects": "false",
>> "osd_max_scrubs": "1",
>> "osd_scrub_load_threshold": "0.5",
>> "osd_scrub_min_interval": "300",
>> "osd_scrub_max_interval": "86400",
>> "osd_deep_scrub_interval": "604800",
>> "osd_deep_scrub_stride": "524288",
>> "osd_auto_weight": "false",
>> "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
>> "osd_check_for_log_corruption": "false",
>> "osd_use_stale_snap": "false",
>> "osd_rollback_to_cluster_snap": "",
>> "osd_default_notify_timeout": "30",
>> "osd_kill_backfill_at": "0",
>> "osd_min_pg_log_entries": "1000",
>> "osd_op_complaint_time": "30",
>> "osd_command_max_records": "256",
>> "osd_op_log_threshold": "5",
>> "osd_verify_sparse_read_holes": "false",
>> "osd_debug_drop_ping_probability": "0",
>> "osd_debug_drop_ping_duration": "0",
>> "osd_debug_drop_pg_create_probability": "0",
>> "osd_debug_drop_pg_create_duration": "1",
>> "osd_debug_drop_op_probability": "0",
>> "osd_op_history_size": "20",
>> "osd_op_history_duration": "600",
>> "osd_target_transaction_size": "300",
>> "osd_client_op_priority": "63",
>> "osd_recovery_op_priority": "10",
>> "filestore": "false",
>> "filestore_index_retry_probability": "0",
>> "filestore_debug_omap_check": "false",
>> "filestore_xattr_use_omap": "false",
>> "filestore_max_inline_xattr_size": "512",
>> "filestore_max_inline_xattrs": "2",
>> "filestore_max_sync_interval": "5",
>> "filestore_min_sync_interval": "0.01",
>> "filestore_btrfs_snap": "true",
>> "filestore_btrfs_clone_range": "true",
>> "filestore_fsync_flushes_journal_data": "false",
>> "filestore_fiemap": "false",
>> "filestore_flusher": "true",
>> "filestore_flusher_max_fds": "512",
>> "filestore_flush_min": "65536",
>> "filestore_sync_flush": "false",
>> "filestore_journal_parallel": "false",
>> "filestore_journal_writeahead": "false",
>> "filestore_journal_trailing": "false",
>> "filestore_queue_max_ops": "500",
>> "filestore_queue_max_bytes": "104857600",
>> "filestore_queue_committing_max_ops": "500",
>> "filestore_queue_committing_max_bytes": "104857600",
>> "filestore_op_threads": "2",
>> "filestore_op_thread_timeout": "60",
>> "filestore_op_thread_suicide_timeout": "180",
>> "filestore_commit_timeout": "600",
>> "filestore_fiemap_threshold": "4096",
>> "filestore_merge_threshold": "10",
>> "filestore_split_multiple": "2",
>> "filestore_update_to": "1000",
>> "filestore_blackhole": "false",
>> "filestore_dump_file": "",
>> "filestore_kill_at": "0",
>> "filestore_fail_eio": "true",
>> "journal_dio": "true",
>> "journal_aio": "false",
>> "journal_block_align": "true",
>> "journal_max_write_bytes": "10485760",
>> "journal_max_write_entries": "100",
>> "journal_queue_max_ops": "500",
>> "journal_queue_max_bytes": "104857600",
>> "journal_align_min_size": "65536",
>> "journal_replay_from": "0",
>> "journal_zero_on_create": "false",
>> "rbd_cache": "false",
>> "rbd_cache_size": "33554432",
>> "rbd_cache_max_dirty": "25165824",
>> "rbd_cache_target_dirty": "16777216",
>> "rbd_cache_max_dirty_age": "1",
>> "nss_db_path": "",
>> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
>> "rgw_enable_apis": "s3, swift, swift_auth, admin",
>> "rgw_cache_enabled": "true",
>> "rgw_cache_lru_size": "10000",
>> "rgw_socket_path": "",
>> "rgw_dns_name": "",
>> "rgw_swift_url": "",
>> "rgw_swift_url_prefix": "swift",
>> "rgw_swift_auth_url": "",
>> "rgw_swift_auth_entry": "auth",
>> "rgw_keystone_url": "",
>> "rgw_keystone_admin_token": "",
>> "rgw_keystone_accepted_roles": "Member, admin",
>> "rgw_keystone_token_cache_size": "10000",
>> "rgw_keystone_revocation_interval": "900",
>> "rgw_admin_entry": "admin",
>> "rgw_enforce_swift_acls": "true",
>> "rgw_swift_token_expiration": "86400",
>> "rgw_print_continue": "true",
>> "rgw_remote_addr_param": "REMOTE_ADDR",
>> "rgw_op_thread_timeout": "600",
>> "rgw_op_thread_suicide_timeout": "0",
>> "rgw_thread_pool_size": "100",
>> "rgw_num_control_oids": "8",
>> "rgw_cluster_root_pool": ".rgw.root",
>> "rgw_log_nonexistent_bucket": "false",
>> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
>> "rgw_log_object_name_utc": "false",
>> "rgw_usage_max_shards": "32",
>> "rgw_usage_max_user_shards": "1",
>> "rgw_enable_ops_log": "true",
>> "rgw_enable_usage_log": "true",
>> "rgw_ops_log_rados": "true",
>> "rgw_ops_log_socket_path": "",
>> "rgw_ops_log_data_backlog": "5242880",
>> "rgw_usage_log_flush_threshold": "1024",
>> "rgw_usage_log_tick_interval": "30",
>> "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
>> "rgw_intent_log_object_name_utc": "false",
>> "rgw_init_timeout": "30",
>> "rgw_mime_types_file": "\/etc\/mime.types",
>> "rgw_gc_max_objs": "32",
>> "rgw_gc_obj_min_wait": "7200",
>> "rgw_gc_processor_max_time": "3600",
>> "rgw_gc_processor_period": "3600",
>> "rgw_s3_success_create_obj_status": "0",
>> "rgw_resolve_cname": "false",
>> "rgw_obj_stripe_size": "4194304",
>> "rgw_extended_http_attrs": "",
>> "mutex_perf_counter": "false",
>> "internal_safe_to_start_threads": "true"}
>>
>>
>> -----Original Message-----
>> From: Mark Nelson [mailto:mark.nelson@inktank.com]
>> Sent: Thursday, 20 December 2012 11:28 PM
>> To: Matthew Anderson
>> Cc: ceph-devel@vger.kernel.org
>> Subject: Re: OSD's slow down to a crawl
>>
>> Hi Matt,
>>
>> Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
>> Sebastien Han made a nice little blog entry describing it:
>>
>> http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
>>
>> You can get a lot of information about the current state of things.
>>
>> Mark
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-20 16:08 ` Matthew Anderson
2012-12-20 16:22 ` Mark Nelson
@ 2012-12-20 16:30 ` Sage Weil
[not found] ` <38A500831D3DE24B90BD200D6C8701351BB15433@Exchange2010-2.corit.local>
1 sibling, 1 reply; 13+ messages in thread
From: Sage Weil @ 2012-12-20 16:30 UTC (permalink / raw)
To: Matthew Anderson; +Cc: 'Mark Nelson', ceph-devel@vger.kernel.org
[-- Attachment #1: Type: TEXT/PLAIN, Size: 41033 bytes --]
Can you do a similar test, but with full logging on?
ceph tell osd.0 injectargs '--debug-ms 1 --debug-filestore 20 --debug-osd
20 --debug-journal 20'
rados -p ssd bench 30 write -b 4096 -t 1 --log-file /tmp/foo --debug-ms 1
That will be a single IO in flight at a time and very easy to trace
through the logs. If you can post the resulting log file (/tmp/foo and
from osd.0), that would be awesome.
Thanks!
sage
On Thu, 20 Dec 2012, Matthew Anderson wrote:
> # rados bench 60 write -t 256 -p ssd
> Maintaining 256 concurrent writes of 4194304 bytes for at least 60 seconds.
> Object prefix: benchmark_data_KVM03_12985
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 45 45 0 0 0 - 0
> 2 71 71 0 0 0 - 0
> 3 90 90 0 0 0 - 0
> 4 125 125 0 0 0 - 0
> 5 136 136 0 0 0 - 0
> 6 144 144 0 0 0 - 0
> 7 175 175 0 0 0 - 0
> 8 201 201 0 0 0 - 0
> 9 217 217 0 0 0 - 0
> 10 228 228 0 0 0 - 0
> 11 251 251 0 0 0 - 0
> 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> 13 255 271 16 4.92194 20 12.9795 11.9979
> 14 255 291 36 10.2833 80 13.1795 12.6478
> 15 255 301 46 12.2639 40 13.9424 12.9123
> 16 255 303 48 11.9973 8 14.0277 12.9579
> 17 255 321 66 15.526 72 15.1408 13.4616
> 18 255 345 90 19.9956 96 15.5673 13.9937
> 19 255 349 94 19.7851 16 15.707 14.0635
> 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 20 255 361 106 21.1953 48 16.4053 14.3121
> 21 255 367 112 21.3285 24 17.4263 14.4795
> 22 255 370 115 20.9043 12 17.3579 14.5544
> 23 255 387 132 22.9513 68 18.3185 15.0867
> 24 255 394 139 23.1614 28 18.2868 15.2504
> 25 255 404 149 23.8346 40 18.741 15.5048
> 26 255 410 155 23.8407 24 19.032 15.6358
> 27 255 410 155 22.9577 0 - 15.6358
> 28 255 410 155 22.1378 0 - 15.6358
> 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> 30 255 431 176 23.4614 80 22.6577 16.4822
> 31 255 447 192 24.7687 64 23.2345 17.0563
> 32 255 464 209 26.1192 68 23.3788 17.5859
> 33 255 485 230 27.8726 84 22.8924 18.1122
> 34 255 497 242 28.4642 48 23.563 18.3693
> 35 255 508 253 28.9078 44 23.0516 18.5935
> 36 255 514 259 28.7713 24 24.4415 18.7278
> 37 255 530 275 29.723 64 23.847 19.0424
> 38 255 541 286 30.0985 44 24.5757 19.2427
> 39 255 544 289 29.6344 12 24.6802 19.2984
> 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 40 255 544 289 28.8935 0 - 19.2984
> 41 255 544 289 28.1888 0 - 19.2984
> 42 255 544 289 27.5176 0 - 19.2984
> 43 255 544 289 26.8776 0 - 19.2984
> 44 255 544 289 26.2668 0 - 19.2984
> 45 255 544 289 25.6831 0 - 19.2984
> 46 255 544 289 25.1248 0 - 19.2984
> 47 255 544 289 24.5902 0 - 19.2984
> 48 255 544 289 24.0779 0 - 19.2984
> 49 255 544 289 23.5865 0 - 19.2984
> 50 255 544 289 23.1148 0 - 19.2984
> 51 255 544 289 22.6616 0 - 19.2984
> 52 255 544 289 22.2258 0 - 19.2984
> 53 255 544 289 21.8065 0 - 19.2984
> 54 255 544 289 21.4026 0 - 19.2984
> 55 255 544 289 21.0135 0 - 19.2984
> 56 255 544 289 20.6383 0 - 19.2984
> 57 255 544 289 20.2762 0 - 19.2984
> 58 255 544 289 19.9266 0 - 19.2984
> 59 255 544 289 19.5889 0 - 19.2984
> 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 60 255 544 289 19.2624 0 - 19.2984
> 61 255 544 289 18.9466 0 - 19.2984
> 62 255 544 289 18.641 0 - 19.2984
> 63 255 544 289 18.3452 0 - 19.2984
> 64 255 544 289 18.0585 0 - 19.2984
> 65 255 544 289 17.7807 0 - 19.2984
> 66 255 544 289 17.5113 0 - 19.2984
> 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> Total time run: 68.000740
> Total writes made: 545
> Write size: 4194304
> Bandwidth (MB/sec): 32.058
>
> Stddev Bandwidth: 26.8837
> Max bandwidth (MB/sec): 96
> Min bandwidth (MB/sec): 0
> Average Latency: 29.2801
> Stddev Latency: 11.9768
> Max latency: 52.6277
> Min latency: 11.4904
>
>
>
>
>
> ----Original Message-----
> From: Mark Nelson [mailto:mark.nelson@inktank.com]
> Sent: Thursday, 20 December 2012 11:59 PM
> To: Matthew Anderson
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: OSD's slow down to a crawl
>
> Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
>
> Mark
>
> On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> > Hi Mark,
> >
> > Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
> >
> > It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
> >
> > The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> > # ceph --admin-daemon ./ceph-osd.24.asok perf dump
> >
> > {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"journ
> > al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_bytes
> > ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"su
> > m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":13
> > 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":20
> > 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":17102325
> > 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
> > :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":1708
> > .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000},
> > "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"avgc
> > ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
> > :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock":{"w
> > ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_lock
> > ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
> > apply_lo
> ck":{"wai
> t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_lock":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_wip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op_latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out_bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000},"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000},"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgcount":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcount":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"subop_push_
> in_bytes"
> :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"push":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"numpg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_from_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epoch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":{"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put_sum":7955
> 46573,"wa
> it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"get":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":199092,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"put_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
> >
> > # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
> >
> > { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> > "received_at": "2012-12-20 23:34:42.012131",
> > "age": "75.428051",
> > "duration": "0.009988",
> > "flag_point": "waiting for sub ops",
> > "client_info": { "client": "client.7636",
> > "tid": 101498},
> > "events": [
> > { "time": "2012-12-20 23:34:42.012317",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:34:42.012425",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:34:42.012678",
> > "event": "started"},
> > { "time": "2012-12-20 23:34:42.012720",
> > "event": "started"},
> > { "time": "2012-12-20 23:34:42.012815",
> > "event": "sub_op_sent"},
> > { "time": "2012-12-20 23:34:42.012958",
> > "event": "commit_queued_for_journal_write"},
> > { "time": "2012-12-20 23:34:42.013033",
> > "event": "write_thread_in_journal_buffer"},
> > { "time": "2012-12-20 23:34:42.013745",
> > "event": "journaled_completion_queued"},
> > { "time": "2012-12-20 23:34:42.013803",
> > "event": "op_commit"},
> > { "time": "2012-12-20 23:34:42.017414",
> > "event": "sub_op_commit_rec"},
> > { "time": "2012-12-20 23:34:42.022066",
> > "event": "op_applied"},
> > { "time": "2012-12-20 23:34:42.022119",
> > "event": "done"}]},
> > { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > "received_at": "2012-12-20 23:34:42.017048",
> > "age": "75.423134",
> > "duration": "0.000411",
> > "flag_point": "started",
> > "events": [
> > { "time": "2012-12-20 23:34:42.017218",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:34:42.017363",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:34:42.017401",
> > "event": "started"},
> > { "time": "2012-12-20 23:34:42.017459",
> > "event": "done"}]},
> > { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> > "received_at": "2012-12-20 23:34:49.851021",
> > "age": "67.589161",
> > "duration": "0.007114",
> > "flag_point": "waiting for sub ops",
> > "client_info": { "client": "client.7636",
> > "tid": 101502},
> > "events": [
> > { "time": "2012-12-20 23:34:49.851276",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:34:49.851429",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:34:49.851616",
> > "event": "started"},
> > { "time": "2012-12-20 23:34:49.851648",
> > "event": "started"},
> > { "time": "2012-12-20 23:34:49.851712",
> > "event": "sub_op_sent"},
> > { "time": "2012-12-20 23:34:49.851770",
> > "event": "commit_queued_for_journal_write"},
> > { "time": "2012-12-20 23:34:49.851915",
> > "event": "write_thread_in_journal_buffer"},
> > { "time": "2012-12-20 23:34:49.853956",
> > "event": "journaled_completion_queued"},
> > { "time": "2012-12-20 23:34:49.853992",
> > "event": "op_commit"},
> > { "time": "2012-12-20 23:34:49.857016",
> > "event": "sub_op_commit_rec"},
> > { "time": "2012-12-20 23:34:49.858077",
> > "event": "op_applied"},
> > { "time": "2012-12-20 23:34:49.858135",
> > "event": "done"}]},
> > { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> > "received_at": "2012-12-20 23:34:49.856762",
> > "age": "67.583420",
> > "duration": "0.000308",
> > "flag_point": "started",
> > "events": [
> > { "time": "2012-12-20 23:34:49.856869",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:34:49.856969",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:34:49.857001",
> > "event": "started"},
> > { "time": "2012-12-20 23:34:49.857070",
> > "event": "done"}]},
> > { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> > "received_at": "2012-12-20 23:35:34.351518",
> > "age": "23.088664",
> > "duration": "0.009430",
> > "flag_point": "waiting for sub ops",
> > "client_info": { "client": "client.7636",
> > "tid": 101504},
> > "events": [
> > { "time": "2012-12-20 23:35:34.351714",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:35:34.351811",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:35:34.352030",
> > "event": "started"},
> > { "time": "2012-12-20 23:35:34.352071",
> > "event": "started"},
> > { "time": "2012-12-20 23:35:34.352146",
> > "event": "sub_op_sent"},
> > { "time": "2012-12-20 23:35:34.352213",
> > "event": "commit_queued_for_journal_write"},
> > { "time": "2012-12-20 23:35:34.352287",
> > "event": "write_thread_in_journal_buffer"},
> > { "time": "2012-12-20 23:35:34.355390",
> > "event": "journaled_completion_queued"},
> > { "time": "2012-12-20 23:35:34.355475",
> > "event": "op_commit"},
> > { "time": "2012-12-20 23:35:34.356661",
> > "event": "sub_op_commit_rec"},
> > { "time": "2012-12-20 23:35:34.360894",
> > "event": "op_applied"},
> > { "time": "2012-12-20 23:35:34.360948",
> > "event": "done"}]},
> > { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> > "received_at": "2012-12-20 23:35:34.356286",
> > "age": "23.083896",
> > "duration": "0.000421",
> > "flag_point": "started",
> > "events": [
> > { "time": "2012-12-20 23:35:34.356489",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:35:34.356598",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:35:34.356640",
> > "event": "started"},
> > { "time": "2012-12-20 23:35:34.356707",
> > "event": "done"}]},
> > { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> > "received_at": "2012-12-20 23:35:42.133103",
> > "age": "15.307079",
> > "duration": "0.008357",
> > "flag_point": "waiting for sub ops",
> > "client_info": { "client": "client.7636",
> > "tid": 101512},
> > "events": [
> > { "time": "2012-12-20 23:35:42.133315",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:35:42.133427",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:35:42.133684",
> > "event": "started"},
> > { "time": "2012-12-20 23:35:42.133723",
> > "event": "started"},
> > { "time": "2012-12-20 23:35:42.133796",
> > "event": "sub_op_sent"},
> > { "time": "2012-12-20 23:35:42.133908",
> > "event": "commit_queued_for_journal_write"},
> > { "time": "2012-12-20 23:35:42.133995",
> > "event": "write_thread_in_journal_buffer"},
> > { "time": "2012-12-20 23:35:42.136965",
> > "event": "journaled_completion_queued"},
> > { "time": "2012-12-20 23:35:42.137025",
> > "event": "op_commit"},
> > { "time": "2012-12-20 23:35:42.138456",
> > "event": "sub_op_commit_rec"},
> > { "time": "2012-12-20 23:35:42.141408",
> > "event": "op_applied"},
> > { "time": "2012-12-20 23:35:42.141460",
> > "event": "done"}]},
> > { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > "received_at": "2012-12-20 23:35:42.138087",
> > "age": "15.302095",
> > "duration": "0.000410",
> > "flag_point": "started",
> > "events": [
> > { "time": "2012-12-20 23:35:42.138248",
> > "event": "waiting_for_osdmap"},
> > { "time": "2012-12-20 23:35:42.138363",
> > "event": "reached_pg"},
> > { "time": "2012-12-20 23:35:42.138443",
> > "event": "started"},
> > { "time": "2012-12-20 23:35:42.138497",
> > "event": "done"}]}]}
> >
> > # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
> > "osd.24",
> > "cluster": "ceph",
> > "none": "0\/5",
> > "lockdep": "0\/1",
> > "context": "0\/1",
> > "crush": "1\/1",
> > "mds": "1\/5",
> > "mds_balancer": "1\/5",
> > "mds_locker": "1\/5",
> > "mds_log": "1\/5",
> > "mds_log_expire": "1\/5",
> > "mds_migrator": "1\/5",
> > "buffer": "0\/1",
> > "timer": "0\/1",
> > "filer": "0\/1",
> > "striper": "0\/1",
> > "objecter": "0\/1",
> > "rados": "0\/5",
> > "rbd": "0\/5",
> > "journaler": "0\/5",
> > "objectcacher": "0\/5",
> > "client": "0\/5",
> > "osd": "0\/5",
> > "optracker": "0\/5",
> > "objclass": "0\/5",
> > "filestore": "1\/3",
> > "journal": "1\/3",
> > "ms": "0\/5",
> > "mon": "1\/5",
> > "monc": "0\/10",
> > "paxos": "0\/5",
> > "tp": "0\/5",
> > "auth": "1\/5",
> > "crypto": "1\/5",
> > "finisher": "1\/1",
> > "heartbeatmap": "1\/5",
> > "perfcounter": "1\/5",
> > "rgw": "1\/5",
> > "hadoop": "1\/5",
> > "javaclient": "1\/5",
> > "asok": "1\/5",
> > "throttle": "1\/1",
> > "host": "KVM04",
> > "fsid": "00000000-0000-0000-0000-000000000000",
> > "public_addr": ":\/0",
> > "cluster_addr": ":\/0",
> > "public_network": "",
> > "cluster_network": "",
> > "num_client": "1",
> > "monmap": "",
> > "mon_host": "",
> > "lockdep": "false",
> > "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> > "daemonize": "true",
> > "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> > "chdir": "\/",
> > "max_open_files": "0",
> > "fatal_signal_handlers": "true",
> > "log_file": "\/var\/log\/ceph\/osd.24.log",
> > "log_max_new": "1000",
> > "log_max_recent": "100000",
> > "log_to_stderr": "false",
> > "err_to_stderr": "true",
> > "log_to_syslog": "false",
> > "err_to_syslog": "false",
> > "log_flush_on_exit": "true",
> > "clog_to_monitors": "true",
> > "clog_to_syslog": "false",
> > "mon_cluster_log_to_syslog": "false",
> > "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> > "key": "",
> > "keyfile": "",
> > "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> > "heartbeat_interval": "5",
> > "heartbeat_file": "",
> > "perf": "true",
> > "ms_tcp_nodelay": "true",
> > "ms_initial_backoff": "0.2",
> > "ms_max_backoff": "15",
> > "ms_nocrc": "false",
> > "ms_die_on_bad_msg": "false",
> > "ms_dispatch_throttle_bytes": "104857600",
> > "ms_bind_ipv6": "false",
> > "ms_bind_port_min": "6800",
> > "ms_bind_port_max": "7100",
> > "ms_rwthread_stack_bytes": "1048576",
> > "ms_tcp_read_timeout": "900",
> > "ms_inject_socket_failures": "0",
> > "ms_inject_delay_type": "",
> > "ms_inject_delay_max": "1",
> > "ms_inject_delay_probability": "0",
> > "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> > "mon_initial_members": "",
> > "mon_sync_fs_threshold": "5",
> > "mon_tick_interval": "5",
> > "mon_subscribe_interval": "300",
> > "mon_osd_laggy_halflife": "3600",
> > "mon_osd_laggy_weight": "0.3",
> > "mon_osd_adjust_heartbeat_grace": "true",
> > "mon_osd_adjust_down_out_interval": "true",
> > "mon_osd_auto_mark_in": "false",
> > "mon_osd_auto_mark_auto_out_in": "true",
> > "mon_osd_auto_mark_new_in": "true",
> > "mon_osd_down_out_interval": "300",
> > "mon_osd_min_up_ratio": "0.3",
> > "mon_osd_min_in_ratio": "0.3",
> > "mon_lease": "5",
> > "mon_lease_renew_interval": "3",
> > "mon_lease_ack_timeout": "10",
> > "mon_clock_drift_allowed": "0.05",
> > "mon_clock_drift_warn_backoff": "5",
> > "mon_accept_timeout": "10",
> > "mon_pg_create_interval": "30",
> > "mon_pg_stuck_threshold": "300",
> > "mon_osd_full_ratio": "0.95",
> > "mon_osd_nearfull_ratio": "0.85",
> > "mon_globalid_prealloc": "100",
> > "mon_osd_report_timeout": "900",
> > "mon_force_standby_active": "true",
> > "mon_min_osdmap_epochs": "500",
> > "mon_max_pgmap_epochs": "500",
> > "mon_max_log_epochs": "500",
> > "mon_max_osd": "10000",
> > "mon_probe_timeout": "2",
> > "mon_slurp_timeout": "10",
> > "mon_slurp_bytes": "262144",
> > "mon_client_bytes": "104857600",
> > "mon_daemon_bytes": "419430400",
> > "mon_max_log_entries_per_event": "4096",
> > "paxos_max_join_drift": "10",
> > "paxos_propose_interval": "1",
> > "paxos_min_wait": "0.05",
> > "clock_offset": "0",
> > "auth_cluster_required": "none",
> > "auth_service_required": "none",
> > "auth_client_required": "none",
> > "auth_supported": "none",
> > "cephx_require_signatures": "false",
> > "cephx_sign_messages": "true",
> > "auth_mon_ticket_ttl": "43200",
> > "auth_service_ticket_ttl": "3600",
> > "mon_client_hunt_interval": "3",
> > "mon_client_ping_interval": "10",
> > "mon_client_max_log_entries_per_message": "1000",
> > "mon_max_pool_pg_num": "65536",
> > "client_cache_size": "16384",
> > "client_cache_mid": "0.75",
> > "client_use_random_mds": "false",
> > "client_mount_timeout": "30",
> > "client_tick_interval": "1",
> > "client_trace": "",
> > "client_readahead_min": "131072",
> > "client_readahead_max_bytes": "0",
> > "client_readahead_max_periods": "4",
> > "client_snapdir": ".snap",
> > "client_mountpoint": "\/",
> > "client_notify_timeout": "10",
> > "client_caps_release_delay": "5",
> > "client_oc": "true",
> > "client_oc_size": "209715200",
> > "client_oc_max_dirty": "104857600",
> > "client_oc_target_dirty": "8388608",
> > "client_oc_max_dirty_age": "5",
> > "client_oc_max_objects": "1000",
> > "client_debug_force_sync_read": "false",
> > "client_debug_inject_tick_delay": "0",
> > "fuse_use_invalidate_cb": "false",
> > "fuse_big_writes": "true",
> > "fuse_debug": "false",
> > "objecter_tick_interval": "5",
> > "objecter_timeout": "10",
> > "objecter_inflight_op_bytes": "104857600",
> > "objecter_inflight_ops": "1024",
> > "journaler_allow_split_entries": "true",
> > "journaler_write_head_interval": "15",
> > "journaler_prefetch_periods": "10",
> > "journaler_prezero_periods": "5",
> > "journaler_batch_interval": "0.001",
> > "journaler_batch_max": "0",
> > "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> > "mds_max_file_size": "1099511627776",
> > "mds_cache_size": "100000",
> > "mds_cache_mid": "0.7",
> > "mds_mem_max": "1048576",
> > "mds_dir_commit_ratio": "0.5",
> > "mds_dir_max_commit_size": "90",
> > "mds_decay_halflife": "5",
> > "mds_beacon_interval": "4",
> > "mds_beacon_grace": "15",
> > "mds_blacklist_interval": "1440",
> > "mds_session_timeout": "60",
> > "mds_session_autoclose": "300",
> > "mds_reconnect_timeout": "45",
> > "mds_tick_interval": "5",
> > "mds_dirstat_min_interval": "1",
> > "mds_scatter_nudge_interval": "5",
> > "mds_client_prealloc_inos": "1000",
> > "mds_early_reply": "true",
> > "mds_use_tmap": "true",
> > "mds_default_dir_hash": "2",
> > "mds_log": "true",
> > "mds_log_skip_corrupt_events": "false",
> > "mds_log_max_events": "-1",
> > "mds_log_max_segments": "30",
> > "mds_log_max_expiring": "20",
> > "mds_bal_sample_interval": "3",
> > "mds_bal_replicate_threshold": "8000",
> > "mds_bal_unreplicate_threshold": "0",
> > "mds_bal_frag": "false",
> > "mds_bal_split_size": "10000",
> > "mds_bal_split_rd": "25000",
> > "mds_bal_split_wr": "10000",
> > "mds_bal_split_bits": "3",
> > "mds_bal_merge_size": "50",
> > "mds_bal_merge_rd": "1000",
> > "mds_bal_merge_wr": "1000",
> > "mds_bal_interval": "10",
> > "mds_bal_fragment_interval": "5",
> > "mds_bal_idle_threshold": "0",
> > "mds_bal_max": "-1",
> > "mds_bal_max_until": "-1",
> > "mds_bal_mode": "0",
> > "mds_bal_min_rebalance": "0.1",
> > "mds_bal_min_start": "0.2",
> > "mds_bal_need_min": "0.8",
> > "mds_bal_need_max": "1.2",
> > "mds_bal_midchunk": "0.3",
> > "mds_bal_minchunk": "0.001",
> > "mds_bal_target_removal_min": "5",
> > "mds_bal_target_removal_max": "10",
> > "mds_replay_interval": "1",
> > "mds_shutdown_check": "0",
> > "mds_thrash_exports": "0",
> > "mds_thrash_fragments": "0",
> > "mds_dump_cache_on_map": "false",
> > "mds_dump_cache_after_rejoin": "false",
> > "mds_verify_scatter": "false",
> > "mds_debug_scatterstat": "false",
> > "mds_debug_frag": "false",
> > "mds_debug_auth_pins": "false",
> > "mds_debug_subtrees": "false",
> > "mds_kill_mdstable_at": "0",
> > "mds_kill_export_at": "0",
> > "mds_kill_import_at": "0",
> > "mds_kill_link_at": "0",
> > "mds_kill_rename_at": "0",
> > "mds_wipe_sessions": "false",
> > "mds_wipe_ino_prealloc": "false",
> > "mds_skip_ino": "0",
> > "max_mds": "1",
> > "mds_standby_for_name": "",
> > "mds_standby_for_rank": "-1",
> > "mds_standby_replay": "false",
> > "osd_auto_upgrade_tmap": "true",
> > "osd_tmapput_sets_uses_tmap": "false",
> > "osd_max_backfills": "10",
> > "osd_backfill_full_ratio": "0.85",
> > "osd_backfill_retry_interval": "10",
> > "osd_uuid": "00000000-0000-0000-0000-000000000000",
> > "osd_data": "\/srv\/ceph\/osd\/osd.24",
> > "osd_journal": "\/dev\/sda3",
> > "osd_journal_size": "5120",
> > "osd_max_write_size": "90",
> > "osd_max_pgls": "1024",
> > "osd_client_message_size_cap": "524288000",
> > "osd_pg_bits": "6",
> > "osd_pgp_bits": "6",
> > "osd_min_rep": "1",
> > "osd_max_rep": "10",
> > "osd_pool_default_crush_rule": "0",
> > "osd_pool_default_size": "2",
> > "osd_pool_default_min_size": "0",
> > "osd_pool_default_pg_num": "8",
> > "osd_pool_default_pgp_num": "8",
> > "osd_map_dedup": "true",
> > "osd_map_cache_size": "500",
> > "osd_map_message_max": "100",
> > "osd_op_threads": "2",
> > "osd_disk_threads": "1",
> > "osd_recovery_threads": "1",
> > "osd_recover_clone_overlap": "true",
> > "osd_backfill_scan_min": "64",
> > "osd_backfill_scan_max": "512",
> > "osd_op_thread_timeout": "30",
> > "osd_recovery_thread_timeout": "30",
> > "osd_snap_trim_thread_timeout": "3600",
> > "osd_scrub_thread_timeout": "60",
> > "osd_scrub_finalize_thread_timeout": "600",
> > "osd_remove_thread_timeout": "3600",
> > "osd_command_thread_timeout": "600",
> > "osd_age": "0.8",
> > "osd_age_time": "0",
> > "osd_heartbeat_addr": ":\/0",
> > "osd_heartbeat_interval": "6",
> > "osd_heartbeat_grace": "20",
> > "osd_mon_heartbeat_interval": "30",
> > "osd_mon_report_interval_max": "120",
> > "osd_mon_report_interval_min": "5",
> > "osd_mon_ack_timeout": "30",
> > "osd_min_down_reporters": "1",
> > "osd_min_down_reports": "3",
> > "osd_default_data_pool_replay_window": "45",
> > "osd_preserve_trimmed_log": "false",
> > "osd_auto_mark_unfound_lost": "false",
> > "osd_recovery_delay_start": "0",
> > "osd_recovery_max_active": "50",
> > "osd_recovery_max_chunk": "8388608",
> > "osd_recovery_forget_lost_objects": "false",
> > "osd_max_scrubs": "1",
> > "osd_scrub_load_threshold": "0.5",
> > "osd_scrub_min_interval": "300",
> > "osd_scrub_max_interval": "86400",
> > "osd_deep_scrub_interval": "604800",
> > "osd_deep_scrub_stride": "524288",
> > "osd_auto_weight": "false",
> > "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> > "osd_check_for_log_corruption": "false",
> > "osd_use_stale_snap": "false",
> > "osd_rollback_to_cluster_snap": "",
> > "osd_default_notify_timeout": "30",
> > "osd_kill_backfill_at": "0",
> > "osd_min_pg_log_entries": "1000",
> > "osd_op_complaint_time": "30",
> > "osd_command_max_records": "256",
> > "osd_op_log_threshold": "5",
> > "osd_verify_sparse_read_holes": "false",
> > "osd_debug_drop_ping_probability": "0",
> > "osd_debug_drop_ping_duration": "0",
> > "osd_debug_drop_pg_create_probability": "0",
> > "osd_debug_drop_pg_create_duration": "1",
> > "osd_debug_drop_op_probability": "0",
> > "osd_op_history_size": "20",
> > "osd_op_history_duration": "600",
> > "osd_target_transaction_size": "300",
> > "osd_client_op_priority": "63",
> > "osd_recovery_op_priority": "10",
> > "filestore": "false",
> > "filestore_index_retry_probability": "0",
> > "filestore_debug_omap_check": "false",
> > "filestore_xattr_use_omap": "false",
> > "filestore_max_inline_xattr_size": "512",
> > "filestore_max_inline_xattrs": "2",
> > "filestore_max_sync_interval": "5",
> > "filestore_min_sync_interval": "0.01",
> > "filestore_btrfs_snap": "true",
> > "filestore_btrfs_clone_range": "true",
> > "filestore_fsync_flushes_journal_data": "false",
> > "filestore_fiemap": "false",
> > "filestore_flusher": "true",
> > "filestore_flusher_max_fds": "512",
> > "filestore_flush_min": "65536",
> > "filestore_sync_flush": "false",
> > "filestore_journal_parallel": "false",
> > "filestore_journal_writeahead": "false",
> > "filestore_journal_trailing": "false",
> > "filestore_queue_max_ops": "500",
> > "filestore_queue_max_bytes": "104857600",
> > "filestore_queue_committing_max_ops": "500",
> > "filestore_queue_committing_max_bytes": "104857600",
> > "filestore_op_threads": "2",
> > "filestore_op_thread_timeout": "60",
> > "filestore_op_thread_suicide_timeout": "180",
> > "filestore_commit_timeout": "600",
> > "filestore_fiemap_threshold": "4096",
> > "filestore_merge_threshold": "10",
> > "filestore_split_multiple": "2",
> > "filestore_update_to": "1000",
> > "filestore_blackhole": "false",
> > "filestore_dump_file": "",
> > "filestore_kill_at": "0",
> > "filestore_fail_eio": "true",
> > "journal_dio": "true",
> > "journal_aio": "false",
> > "journal_block_align": "true",
> > "journal_max_write_bytes": "10485760",
> > "journal_max_write_entries": "100",
> > "journal_queue_max_ops": "500",
> > "journal_queue_max_bytes": "104857600",
> > "journal_align_min_size": "65536",
> > "journal_replay_from": "0",
> > "journal_zero_on_create": "false",
> > "rbd_cache": "false",
> > "rbd_cache_size": "33554432",
> > "rbd_cache_max_dirty": "25165824",
> > "rbd_cache_target_dirty": "16777216",
> > "rbd_cache_max_dirty_age": "1",
> > "nss_db_path": "",
> > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> > "rgw_enable_apis": "s3, swift, swift_auth, admin",
> > "rgw_cache_enabled": "true",
> > "rgw_cache_lru_size": "10000",
> > "rgw_socket_path": "",
> > "rgw_dns_name": "",
> > "rgw_swift_url": "",
> > "rgw_swift_url_prefix": "swift",
> > "rgw_swift_auth_url": "",
> > "rgw_swift_auth_entry": "auth",
> > "rgw_keystone_url": "",
> > "rgw_keystone_admin_token": "",
> > "rgw_keystone_accepted_roles": "Member, admin",
> > "rgw_keystone_token_cache_size": "10000",
> > "rgw_keystone_revocation_interval": "900",
> > "rgw_admin_entry": "admin",
> > "rgw_enforce_swift_acls": "true",
> > "rgw_swift_token_expiration": "86400",
> > "rgw_print_continue": "true",
> > "rgw_remote_addr_param": "REMOTE_ADDR",
> > "rgw_op_thread_timeout": "600",
> > "rgw_op_thread_suicide_timeout": "0",
> > "rgw_thread_pool_size": "100",
> > "rgw_num_control_oids": "8",
> > "rgw_cluster_root_pool": ".rgw.root",
> > "rgw_log_nonexistent_bucket": "false",
> > "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > "rgw_log_object_name_utc": "false",
> > "rgw_usage_max_shards": "32",
> > "rgw_usage_max_user_shards": "1",
> > "rgw_enable_ops_log": "true",
> > "rgw_enable_usage_log": "true",
> > "rgw_ops_log_rados": "true",
> > "rgw_ops_log_socket_path": "",
> > "rgw_ops_log_data_backlog": "5242880",
> > "rgw_usage_log_flush_threshold": "1024",
> > "rgw_usage_log_tick_interval": "30",
> > "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> > "rgw_intent_log_object_name_utc": "false",
> > "rgw_init_timeout": "30",
> > "rgw_mime_types_file": "\/etc\/mime.types",
> > "rgw_gc_max_objs": "32",
> > "rgw_gc_obj_min_wait": "7200",
> > "rgw_gc_processor_max_time": "3600",
> > "rgw_gc_processor_period": "3600",
> > "rgw_s3_success_create_obj_status": "0",
> > "rgw_resolve_cname": "false",
> > "rgw_obj_stripe_size": "4194304",
> > "rgw_extended_http_attrs": "",
> > "mutex_perf_counter": "false",
> > "internal_safe_to_start_threads": "true"}
> >
> >
> > -----Original Message-----
> > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > Sent: Thursday, 20 December 2012 11:28 PM
> > To: Matthew Anderson
> > Cc: ceph-devel@vger.kernel.org
> > Subject: Re: OSD's slow down to a crawl
> >
> > Hi Matt,
> >
> > Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> > Sebastien Han made a nice little blog entry describing it:
> >
> > http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
> >
> > You can get a lot of information about the current state of things.
> >
> > Mark
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-20 16:22 ` Mark Nelson
@ 2012-12-20 16:34 ` Matthew Anderson
0 siblings, 0 replies; 13+ messages in thread
From: Matthew Anderson @ 2012-12-20 16:34 UTC (permalink / raw)
To: 'Mark Nelson'; +Cc: ceph-devel@vger.kernel.org
In regards to syncfs, I'm running Scientific Linux 6.3 which has glibc 2.12 installed. The docs do mention that 0.55 and beyond doesn't require glibc syncfs support, only kernel support. I thought I would mention it just in case it was worth recompiling with a newer version of glibc.
Networking is IPoIB using the supplied kernel modules and Mellanox Connect-X2 QDR cards. I've tested in both datagram and connected modes and there was no difference between them. Iperf shows roughly 13Gbits between hosts and there haven't been any packet drops or errors reported by ifconfig or the kernel stats for the IB card.
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Friday, 21 December 2012 12:22 AM
To: Matthew Anderson
Cc: ceph-devel@vger.kernel.org
Subject: Re: OSD's slow down to a crawl
Ok,
So the interesting news here is that the OSDs don't appear to be stalled exactly, but they do seem to be going really slow. With that new of a kernel and 0.55 you should have syncfs support so I don't think that's the problem. What kind of networking? Any possible problems there?
Mark
On 12/20/2012 10:08 AM, Matthew Anderson wrote:
> # rados bench 60 write -t 256 -p ssd
> Maintaining 256 concurrent writes of 4194304 bytes for at least 60 seconds.
> Object prefix: benchmark_data_KVM03_12985
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 0 0 0 0 0 - 0
> 1 45 45 0 0 0 - 0
> 2 71 71 0 0 0 - 0
> 3 90 90 0 0 0 - 0
> 4 125 125 0 0 0 - 0
> 5 136 136 0 0 0 - 0
> 6 144 144 0 0 0 - 0
> 7 175 175 0 0 0 - 0
> 8 201 201 0 0 0 - 0
> 9 217 217 0 0 0 - 0
> 10 228 228 0 0 0 - 0
> 11 251 251 0 0 0 - 0
> 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> 13 255 271 16 4.92194 20 12.9795 11.9979
> 14 255 291 36 10.2833 80 13.1795 12.6478
> 15 255 301 46 12.2639 40 13.9424 12.9123
> 16 255 303 48 11.9973 8 14.0277 12.9579
> 17 255 321 66 15.526 72 15.1408 13.4616
> 18 255 345 90 19.9956 96 15.5673 13.9937
> 19 255 349 94 19.7851 16 15.707 14.0635
> 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 20 255 361 106 21.1953 48 16.4053 14.3121
> 21 255 367 112 21.3285 24 17.4263 14.4795
> 22 255 370 115 20.9043 12 17.3579 14.5544
> 23 255 387 132 22.9513 68 18.3185 15.0867
> 24 255 394 139 23.1614 28 18.2868 15.2504
> 25 255 404 149 23.8346 40 18.741 15.5048
> 26 255 410 155 23.8407 24 19.032 15.6358
> 27 255 410 155 22.9577 0 - 15.6358
> 28 255 410 155 22.1378 0 - 15.6358
> 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> 30 255 431 176 23.4614 80 22.6577 16.4822
> 31 255 447 192 24.7687 64 23.2345 17.0563
> 32 255 464 209 26.1192 68 23.3788 17.5859
> 33 255 485 230 27.8726 84 22.8924 18.1122
> 34 255 497 242 28.4642 48 23.563 18.3693
> 35 255 508 253 28.9078 44 23.0516 18.5935
> 36 255 514 259 28.7713 24 24.4415 18.7278
> 37 255 530 275 29.723 64 23.847 19.0424
> 38 255 541 286 30.0985 44 24.5757 19.2427
> 39 255 544 289 29.6344 12 24.6802 19.2984
> 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 40 255 544 289 28.8935 0 - 19.2984
> 41 255 544 289 28.1888 0 - 19.2984
> 42 255 544 289 27.5176 0 - 19.2984
> 43 255 544 289 26.8776 0 - 19.2984
> 44 255 544 289 26.2668 0 - 19.2984
> 45 255 544 289 25.6831 0 - 19.2984
> 46 255 544 289 25.1248 0 - 19.2984
> 47 255 544 289 24.5902 0 - 19.2984
> 48 255 544 289 24.0779 0 - 19.2984
> 49 255 544 289 23.5865 0 - 19.2984
> 50 255 544 289 23.1148 0 - 19.2984
> 51 255 544 289 22.6616 0 - 19.2984
> 52 255 544 289 22.2258 0 - 19.2984
> 53 255 544 289 21.8065 0 - 19.2984
> 54 255 544 289 21.4026 0 - 19.2984
> 55 255 544 289 21.0135 0 - 19.2984
> 56 255 544 289 20.6383 0 - 19.2984
> 57 255 544 289 20.2762 0 - 19.2984
> 58 255 544 289 19.9266 0 - 19.2984
> 59 255 544 289 19.5889 0 - 19.2984
> 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 60 255 544 289 19.2624 0 - 19.2984
> 61 255 544 289 18.9466 0 - 19.2984
> 62 255 544 289 18.641 0 - 19.2984
> 63 255 544 289 18.3452 0 - 19.2984
> 64 255 544 289 18.0585 0 - 19.2984
> 65 255 544 289 17.7807 0 - 19.2984
> 66 255 544 289 17.5113 0 - 19.2984
> 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> Total time run: 68.000740
> Total writes made: 545
> Write size: 4194304
> Bandwidth (MB/sec): 32.058
>
> Stddev Bandwidth: 26.8837
> Max bandwidth (MB/sec): 96
> Min bandwidth (MB/sec): 0
> Average Latency: 29.2801
> Stddev Latency: 11.9768
> Max latency: 52.6277
> Min latency: 11.4904
>
>
>
>
>
> ----Original Message-----
> From: Mark Nelson [mailto:mark.nelson@inktank.com]
> Sent: Thursday, 20 December 2012 11:59 PM
> To: Matthew Anderson
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: OSD's slow down to a crawl
>
> Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
>
> Mark
>
> On 12/20/2012 09:51 AM, Matthew Anderson wrote:
>> Hi Mark,
>>
>> Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
>>
>> It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
>>
>> The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
>> # ceph --admin-daemon ./ceph-osd.24.asok perf dump
>>
>> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"jour
>> n
>> al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_byte
>> s
>> ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"s
>> u
>> m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":1
>> 3
>> 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":2
>> 0
>> 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1710232
>> 5
>> 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
>> :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":170
>> 8
>> .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000}
>> ,
>> "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"avg
>> c
>> ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
>> :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock":{"
>> w
>> ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_loc
>> k
>> ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
>> apply_lo
> ck":{"wai
> t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_loc
> k":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager
> ::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_w
> ip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op
> _latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out
> _bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000
> },"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"
> sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000}
> ,"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgco
> unt":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000
> 000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcoun
> t":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"
> subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"
> subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"s
> ubop_pus
h_
> in_bytes"
> :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"pus
> h":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0
> ,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"num
> pg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_fro
> m_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epo
> ch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get
> ":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20
> 868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"av
> gcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":
> 500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"
> take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgc
> ount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":
> {"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_
> fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put
> _sum":79
55
> 46573,"wa
> it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttle
> r-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"g
> et_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put"
> :18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"t
> hrottle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"ge
> t":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,
> "take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0
> ,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val
> ":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"
> get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":1990
> 92,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes
> ":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_fai
> l_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"p
> ut_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
>>
>> # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
>>
>> { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
>> "received_at": "2012-12-20 23:34:42.012131",
>> "age": "75.428051",
>> "duration": "0.009988",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101498},
>> "events": [
>> { "time": "2012-12-20 23:34:42.012317",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:42.012425",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:42.012678",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:42.012720",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:42.012815",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:34:42.012958",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:34:42.013033",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:34:42.013745",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:34:42.013803",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:34:42.017414",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:34:42.022066",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:34:42.022119",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:34:42.017048",
>> "age": "75.423134",
>> "duration": "0.000411",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:34:42.017218",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:42.017363",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:42.017401",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:42.017459",
>> "event": "done"}]},
>> { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
>> "received_at": "2012-12-20 23:34:49.851021",
>> "age": "67.589161",
>> "duration": "0.007114",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101502},
>> "events": [
>> { "time": "2012-12-20 23:34:49.851276",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:49.851429",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:49.851616",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:49.851648",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:49.851712",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:34:49.851770",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:34:49.851915",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:34:49.853956",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:34:49.853992",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:34:49.857016",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:34:49.858077",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:34:49.858135",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:34:49.856762",
>> "age": "67.583420",
>> "duration": "0.000308",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:34:49.856869",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:34:49.856969",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:34:49.857001",
>> "event": "started"},
>> { "time": "2012-12-20 23:34:49.857070",
>> "event": "done"}]},
>> { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
>> "received_at": "2012-12-20 23:35:34.351518",
>> "age": "23.088664",
>> "duration": "0.009430",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101504},
>> "events": [
>> { "time": "2012-12-20 23:35:34.351714",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:34.351811",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:34.352030",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:34.352071",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:34.352146",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:35:34.352213",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:35:34.352287",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:35:34.355390",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:35:34.355475",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:35:34.356661",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:35:34.360894",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:35:34.360948",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:35:34.356286",
>> "age": "23.083896",
>> "duration": "0.000421",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:35:34.356489",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:34.356598",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:34.356640",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:34.356707",
>> "event": "done"}]},
>> { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
>> "received_at": "2012-12-20 23:35:42.133103",
>> "age": "15.307079",
>> "duration": "0.008357",
>> "flag_point": "waiting for sub ops",
>> "client_info": { "client": "client.7636",
>> "tid": 101512},
>> "events": [
>> { "time": "2012-12-20 23:35:42.133315",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:42.133427",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:42.133684",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:42.133723",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:42.133796",
>> "event": "sub_op_sent"},
>> { "time": "2012-12-20 23:35:42.133908",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "2012-12-20 23:35:42.133995",
>> "event": "write_thread_in_journal_buffer"},
>> { "time": "2012-12-20 23:35:42.136965",
>> "event": "journaled_completion_queued"},
>> { "time": "2012-12-20 23:35:42.137025",
>> "event": "op_commit"},
>> { "time": "2012-12-20 23:35:42.138456",
>> "event": "sub_op_commit_rec"},
>> { "time": "2012-12-20 23:35:42.141408",
>> "event": "op_applied"},
>> { "time": "2012-12-20 23:35:42.141460",
>> "event": "done"}]},
>> { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
>> "received_at": "2012-12-20 23:35:42.138087",
>> "age": "15.302095",
>> "duration": "0.000410",
>> "flag_point": "started",
>> "events": [
>> { "time": "2012-12-20 23:35:42.138248",
>> "event": "waiting_for_osdmap"},
>> { "time": "2012-12-20 23:35:42.138363",
>> "event": "reached_pg"},
>> { "time": "2012-12-20 23:35:42.138443",
>> "event": "started"},
>> { "time": "2012-12-20 23:35:42.138497",
>> "event": "done"}]}]}
>>
>> # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
>> "osd.24",
>> "cluster": "ceph",
>> "none": "0\/5",
>> "lockdep": "0\/1",
>> "context": "0\/1",
>> "crush": "1\/1",
>> "mds": "1\/5",
>> "mds_balancer": "1\/5",
>> "mds_locker": "1\/5",
>> "mds_log": "1\/5",
>> "mds_log_expire": "1\/5",
>> "mds_migrator": "1\/5",
>> "buffer": "0\/1",
>> "timer": "0\/1",
>> "filer": "0\/1",
>> "striper": "0\/1",
>> "objecter": "0\/1",
>> "rados": "0\/5",
>> "rbd": "0\/5",
>> "journaler": "0\/5",
>> "objectcacher": "0\/5",
>> "client": "0\/5",
>> "osd": "0\/5",
>> "optracker": "0\/5",
>> "objclass": "0\/5",
>> "filestore": "1\/3",
>> "journal": "1\/3",
>> "ms": "0\/5",
>> "mon": "1\/5",
>> "monc": "0\/10",
>> "paxos": "0\/5",
>> "tp": "0\/5",
>> "auth": "1\/5",
>> "crypto": "1\/5",
>> "finisher": "1\/1",
>> "heartbeatmap": "1\/5",
>> "perfcounter": "1\/5",
>> "rgw": "1\/5",
>> "hadoop": "1\/5",
>> "javaclient": "1\/5",
>> "asok": "1\/5",
>> "throttle": "1\/1",
>> "host": "KVM04",
>> "fsid": "00000000-0000-0000-0000-000000000000",
>> "public_addr": ":\/0",
>> "cluster_addr": ":\/0",
>> "public_network": "",
>> "cluster_network": "",
>> "num_client": "1",
>> "monmap": "",
>> "mon_host": "",
>> "lockdep": "false",
>> "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
>> "daemonize": "true",
>> "pid_file": "\/var\/run\/ceph\/osd.24.pid",
>> "chdir": "\/",
>> "max_open_files": "0",
>> "fatal_signal_handlers": "true",
>> "log_file": "\/var\/log\/ceph\/osd.24.log",
>> "log_max_new": "1000",
>> "log_max_recent": "100000",
>> "log_to_stderr": "false",
>> "err_to_stderr": "true",
>> "log_to_syslog": "false",
>> "err_to_syslog": "false",
>> "log_flush_on_exit": "true",
>> "clog_to_monitors": "true",
>> "clog_to_syslog": "false",
>> "mon_cluster_log_to_syslog": "false",
>> "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
>> "key": "",
>> "keyfile": "",
>> "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
>> "heartbeat_interval": "5",
>> "heartbeat_file": "",
>> "perf": "true",
>> "ms_tcp_nodelay": "true",
>> "ms_initial_backoff": "0.2",
>> "ms_max_backoff": "15",
>> "ms_nocrc": "false",
>> "ms_die_on_bad_msg": "false",
>> "ms_dispatch_throttle_bytes": "104857600",
>> "ms_bind_ipv6": "false",
>> "ms_bind_port_min": "6800",
>> "ms_bind_port_max": "7100",
>> "ms_rwthread_stack_bytes": "1048576",
>> "ms_tcp_read_timeout": "900",
>> "ms_inject_socket_failures": "0",
>> "ms_inject_delay_type": "",
>> "ms_inject_delay_max": "1",
>> "ms_inject_delay_probability": "0",
>> "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
>> "mon_initial_members": "",
>> "mon_sync_fs_threshold": "5",
>> "mon_tick_interval": "5",
>> "mon_subscribe_interval": "300",
>> "mon_osd_laggy_halflife": "3600",
>> "mon_osd_laggy_weight": "0.3",
>> "mon_osd_adjust_heartbeat_grace": "true",
>> "mon_osd_adjust_down_out_interval": "true",
>> "mon_osd_auto_mark_in": "false",
>> "mon_osd_auto_mark_auto_out_in": "true",
>> "mon_osd_auto_mark_new_in": "true",
>> "mon_osd_down_out_interval": "300",
>> "mon_osd_min_up_ratio": "0.3",
>> "mon_osd_min_in_ratio": "0.3",
>> "mon_lease": "5",
>> "mon_lease_renew_interval": "3",
>> "mon_lease_ack_timeout": "10",
>> "mon_clock_drift_allowed": "0.05",
>> "mon_clock_drift_warn_backoff": "5",
>> "mon_accept_timeout": "10",
>> "mon_pg_create_interval": "30",
>> "mon_pg_stuck_threshold": "300",
>> "mon_osd_full_ratio": "0.95",
>> "mon_osd_nearfull_ratio": "0.85",
>> "mon_globalid_prealloc": "100",
>> "mon_osd_report_timeout": "900",
>> "mon_force_standby_active": "true",
>> "mon_min_osdmap_epochs": "500",
>> "mon_max_pgmap_epochs": "500",
>> "mon_max_log_epochs": "500",
>> "mon_max_osd": "10000",
>> "mon_probe_timeout": "2",
>> "mon_slurp_timeout": "10",
>> "mon_slurp_bytes": "262144",
>> "mon_client_bytes": "104857600",
>> "mon_daemon_bytes": "419430400",
>> "mon_max_log_entries_per_event": "4096",
>> "paxos_max_join_drift": "10",
>> "paxos_propose_interval": "1",
>> "paxos_min_wait": "0.05",
>> "clock_offset": "0",
>> "auth_cluster_required": "none",
>> "auth_service_required": "none",
>> "auth_client_required": "none",
>> "auth_supported": "none",
>> "cephx_require_signatures": "false",
>> "cephx_sign_messages": "true",
>> "auth_mon_ticket_ttl": "43200",
>> "auth_service_ticket_ttl": "3600",
>> "mon_client_hunt_interval": "3",
>> "mon_client_ping_interval": "10",
>> "mon_client_max_log_entries_per_message": "1000",
>> "mon_max_pool_pg_num": "65536",
>> "client_cache_size": "16384",
>> "client_cache_mid": "0.75",
>> "client_use_random_mds": "false",
>> "client_mount_timeout": "30",
>> "client_tick_interval": "1",
>> "client_trace": "",
>> "client_readahead_min": "131072",
>> "client_readahead_max_bytes": "0",
>> "client_readahead_max_periods": "4",
>> "client_snapdir": ".snap",
>> "client_mountpoint": "\/",
>> "client_notify_timeout": "10",
>> "client_caps_release_delay": "5",
>> "client_oc": "true",
>> "client_oc_size": "209715200",
>> "client_oc_max_dirty": "104857600",
>> "client_oc_target_dirty": "8388608",
>> "client_oc_max_dirty_age": "5",
>> "client_oc_max_objects": "1000",
>> "client_debug_force_sync_read": "false",
>> "client_debug_inject_tick_delay": "0",
>> "fuse_use_invalidate_cb": "false",
>> "fuse_big_writes": "true",
>> "fuse_debug": "false",
>> "objecter_tick_interval": "5",
>> "objecter_timeout": "10",
>> "objecter_inflight_op_bytes": "104857600",
>> "objecter_inflight_ops": "1024",
>> "journaler_allow_split_entries": "true",
>> "journaler_write_head_interval": "15",
>> "journaler_prefetch_periods": "10",
>> "journaler_prezero_periods": "5",
>> "journaler_batch_interval": "0.001",
>> "journaler_batch_max": "0",
>> "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
>> "mds_max_file_size": "1099511627776",
>> "mds_cache_size": "100000",
>> "mds_cache_mid": "0.7",
>> "mds_mem_max": "1048576",
>> "mds_dir_commit_ratio": "0.5",
>> "mds_dir_max_commit_size": "90",
>> "mds_decay_halflife": "5",
>> "mds_beacon_interval": "4",
>> "mds_beacon_grace": "15",
>> "mds_blacklist_interval": "1440",
>> "mds_session_timeout": "60",
>> "mds_session_autoclose": "300",
>> "mds_reconnect_timeout": "45",
>> "mds_tick_interval": "5",
>> "mds_dirstat_min_interval": "1",
>> "mds_scatter_nudge_interval": "5",
>> "mds_client_prealloc_inos": "1000",
>> "mds_early_reply": "true",
>> "mds_use_tmap": "true",
>> "mds_default_dir_hash": "2",
>> "mds_log": "true",
>> "mds_log_skip_corrupt_events": "false",
>> "mds_log_max_events": "-1",
>> "mds_log_max_segments": "30",
>> "mds_log_max_expiring": "20",
>> "mds_bal_sample_interval": "3",
>> "mds_bal_replicate_threshold": "8000",
>> "mds_bal_unreplicate_threshold": "0",
>> "mds_bal_frag": "false",
>> "mds_bal_split_size": "10000",
>> "mds_bal_split_rd": "25000",
>> "mds_bal_split_wr": "10000",
>> "mds_bal_split_bits": "3",
>> "mds_bal_merge_size": "50",
>> "mds_bal_merge_rd": "1000",
>> "mds_bal_merge_wr": "1000",
>> "mds_bal_interval": "10",
>> "mds_bal_fragment_interval": "5",
>> "mds_bal_idle_threshold": "0",
>> "mds_bal_max": "-1",
>> "mds_bal_max_until": "-1",
>> "mds_bal_mode": "0",
>> "mds_bal_min_rebalance": "0.1",
>> "mds_bal_min_start": "0.2",
>> "mds_bal_need_min": "0.8",
>> "mds_bal_need_max": "1.2",
>> "mds_bal_midchunk": "0.3",
>> "mds_bal_minchunk": "0.001",
>> "mds_bal_target_removal_min": "5",
>> "mds_bal_target_removal_max": "10",
>> "mds_replay_interval": "1",
>> "mds_shutdown_check": "0",
>> "mds_thrash_exports": "0",
>> "mds_thrash_fragments": "0",
>> "mds_dump_cache_on_map": "false",
>> "mds_dump_cache_after_rejoin": "false",
>> "mds_verify_scatter": "false",
>> "mds_debug_scatterstat": "false",
>> "mds_debug_frag": "false",
>> "mds_debug_auth_pins": "false",
>> "mds_debug_subtrees": "false",
>> "mds_kill_mdstable_at": "0",
>> "mds_kill_export_at": "0",
>> "mds_kill_import_at": "0",
>> "mds_kill_link_at": "0",
>> "mds_kill_rename_at": "0",
>> "mds_wipe_sessions": "false",
>> "mds_wipe_ino_prealloc": "false",
>> "mds_skip_ino": "0",
>> "max_mds": "1",
>> "mds_standby_for_name": "",
>> "mds_standby_for_rank": "-1",
>> "mds_standby_replay": "false",
>> "osd_auto_upgrade_tmap": "true",
>> "osd_tmapput_sets_uses_tmap": "false",
>> "osd_max_backfills": "10",
>> "osd_backfill_full_ratio": "0.85",
>> "osd_backfill_retry_interval": "10",
>> "osd_uuid": "00000000-0000-0000-0000-000000000000",
>> "osd_data": "\/srv\/ceph\/osd\/osd.24",
>> "osd_journal": "\/dev\/sda3",
>> "osd_journal_size": "5120",
>> "osd_max_write_size": "90",
>> "osd_max_pgls": "1024",
>> "osd_client_message_size_cap": "524288000",
>> "osd_pg_bits": "6",
>> "osd_pgp_bits": "6",
>> "osd_min_rep": "1",
>> "osd_max_rep": "10",
>> "osd_pool_default_crush_rule": "0",
>> "osd_pool_default_size": "2",
>> "osd_pool_default_min_size": "0",
>> "osd_pool_default_pg_num": "8",
>> "osd_pool_default_pgp_num": "8",
>> "osd_map_dedup": "true",
>> "osd_map_cache_size": "500",
>> "osd_map_message_max": "100",
>> "osd_op_threads": "2",
>> "osd_disk_threads": "1",
>> "osd_recovery_threads": "1",
>> "osd_recover_clone_overlap": "true",
>> "osd_backfill_scan_min": "64",
>> "osd_backfill_scan_max": "512",
>> "osd_op_thread_timeout": "30",
>> "osd_recovery_thread_timeout": "30",
>> "osd_snap_trim_thread_timeout": "3600",
>> "osd_scrub_thread_timeout": "60",
>> "osd_scrub_finalize_thread_timeout": "600",
>> "osd_remove_thread_timeout": "3600",
>> "osd_command_thread_timeout": "600",
>> "osd_age": "0.8",
>> "osd_age_time": "0",
>> "osd_heartbeat_addr": ":\/0",
>> "osd_heartbeat_interval": "6",
>> "osd_heartbeat_grace": "20",
>> "osd_mon_heartbeat_interval": "30",
>> "osd_mon_report_interval_max": "120",
>> "osd_mon_report_interval_min": "5",
>> "osd_mon_ack_timeout": "30",
>> "osd_min_down_reporters": "1",
>> "osd_min_down_reports": "3",
>> "osd_default_data_pool_replay_window": "45",
>> "osd_preserve_trimmed_log": "false",
>> "osd_auto_mark_unfound_lost": "false",
>> "osd_recovery_delay_start": "0",
>> "osd_recovery_max_active": "50",
>> "osd_recovery_max_chunk": "8388608",
>> "osd_recovery_forget_lost_objects": "false",
>> "osd_max_scrubs": "1",
>> "osd_scrub_load_threshold": "0.5",
>> "osd_scrub_min_interval": "300",
>> "osd_scrub_max_interval": "86400",
>> "osd_deep_scrub_interval": "604800",
>> "osd_deep_scrub_stride": "524288",
>> "osd_auto_weight": "false",
>> "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
>> "osd_check_for_log_corruption": "false",
>> "osd_use_stale_snap": "false",
>> "osd_rollback_to_cluster_snap": "",
>> "osd_default_notify_timeout": "30",
>> "osd_kill_backfill_at": "0",
>> "osd_min_pg_log_entries": "1000",
>> "osd_op_complaint_time": "30",
>> "osd_command_max_records": "256",
>> "osd_op_log_threshold": "5",
>> "osd_verify_sparse_read_holes": "false",
>> "osd_debug_drop_ping_probability": "0",
>> "osd_debug_drop_ping_duration": "0",
>> "osd_debug_drop_pg_create_probability": "0",
>> "osd_debug_drop_pg_create_duration": "1",
>> "osd_debug_drop_op_probability": "0",
>> "osd_op_history_size": "20",
>> "osd_op_history_duration": "600",
>> "osd_target_transaction_size": "300",
>> "osd_client_op_priority": "63",
>> "osd_recovery_op_priority": "10",
>> "filestore": "false",
>> "filestore_index_retry_probability": "0",
>> "filestore_debug_omap_check": "false",
>> "filestore_xattr_use_omap": "false",
>> "filestore_max_inline_xattr_size": "512",
>> "filestore_max_inline_xattrs": "2",
>> "filestore_max_sync_interval": "5",
>> "filestore_min_sync_interval": "0.01",
>> "filestore_btrfs_snap": "true",
>> "filestore_btrfs_clone_range": "true",
>> "filestore_fsync_flushes_journal_data": "false",
>> "filestore_fiemap": "false",
>> "filestore_flusher": "true",
>> "filestore_flusher_max_fds": "512",
>> "filestore_flush_min": "65536",
>> "filestore_sync_flush": "false",
>> "filestore_journal_parallel": "false",
>> "filestore_journal_writeahead": "false",
>> "filestore_journal_trailing": "false",
>> "filestore_queue_max_ops": "500",
>> "filestore_queue_max_bytes": "104857600",
>> "filestore_queue_committing_max_ops": "500",
>> "filestore_queue_committing_max_bytes": "104857600",
>> "filestore_op_threads": "2",
>> "filestore_op_thread_timeout": "60",
>> "filestore_op_thread_suicide_timeout": "180",
>> "filestore_commit_timeout": "600",
>> "filestore_fiemap_threshold": "4096",
>> "filestore_merge_threshold": "10",
>> "filestore_split_multiple": "2",
>> "filestore_update_to": "1000",
>> "filestore_blackhole": "false",
>> "filestore_dump_file": "",
>> "filestore_kill_at": "0",
>> "filestore_fail_eio": "true",
>> "journal_dio": "true",
>> "journal_aio": "false",
>> "journal_block_align": "true",
>> "journal_max_write_bytes": "10485760",
>> "journal_max_write_entries": "100",
>> "journal_queue_max_ops": "500",
>> "journal_queue_max_bytes": "104857600",
>> "journal_align_min_size": "65536",
>> "journal_replay_from": "0",
>> "journal_zero_on_create": "false",
>> "rbd_cache": "false",
>> "rbd_cache_size": "33554432",
>> "rbd_cache_max_dirty": "25165824",
>> "rbd_cache_target_dirty": "16777216",
>> "rbd_cache_max_dirty_age": "1",
>> "nss_db_path": "",
>> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
>> "rgw_enable_apis": "s3, swift, swift_auth, admin",
>> "rgw_cache_enabled": "true",
>> "rgw_cache_lru_size": "10000",
>> "rgw_socket_path": "",
>> "rgw_dns_name": "",
>> "rgw_swift_url": "",
>> "rgw_swift_url_prefix": "swift",
>> "rgw_swift_auth_url": "",
>> "rgw_swift_auth_entry": "auth",
>> "rgw_keystone_url": "",
>> "rgw_keystone_admin_token": "",
>> "rgw_keystone_accepted_roles": "Member, admin",
>> "rgw_keystone_token_cache_size": "10000",
>> "rgw_keystone_revocation_interval": "900",
>> "rgw_admin_entry": "admin",
>> "rgw_enforce_swift_acls": "true",
>> "rgw_swift_token_expiration": "86400",
>> "rgw_print_continue": "true",
>> "rgw_remote_addr_param": "REMOTE_ADDR",
>> "rgw_op_thread_timeout": "600",
>> "rgw_op_thread_suicide_timeout": "0",
>> "rgw_thread_pool_size": "100",
>> "rgw_num_control_oids": "8",
>> "rgw_cluster_root_pool": ".rgw.root",
>> "rgw_log_nonexistent_bucket": "false",
>> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
>> "rgw_log_object_name_utc": "false",
>> "rgw_usage_max_shards": "32",
>> "rgw_usage_max_user_shards": "1",
>> "rgw_enable_ops_log": "true",
>> "rgw_enable_usage_log": "true",
>> "rgw_ops_log_rados": "true",
>> "rgw_ops_log_socket_path": "",
>> "rgw_ops_log_data_backlog": "5242880",
>> "rgw_usage_log_flush_threshold": "1024",
>> "rgw_usage_log_tick_interval": "30",
>> "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
>> "rgw_intent_log_object_name_utc": "false",
>> "rgw_init_timeout": "30",
>> "rgw_mime_types_file": "\/etc\/mime.types",
>> "rgw_gc_max_objs": "32",
>> "rgw_gc_obj_min_wait": "7200",
>> "rgw_gc_processor_max_time": "3600",
>> "rgw_gc_processor_period": "3600",
>> "rgw_s3_success_create_obj_status": "0",
>> "rgw_resolve_cname": "false",
>> "rgw_obj_stripe_size": "4194304",
>> "rgw_extended_http_attrs": "",
>> "mutex_perf_counter": "false",
>> "internal_safe_to_start_threads": "true"}
>>
>>
>> -----Original Message-----
>> From: Mark Nelson [mailto:mark.nelson@inktank.com]
>> Sent: Thursday, 20 December 2012 11:28 PM
>> To: Matthew Anderson
>> Cc: ceph-devel@vger.kernel.org
>> Subject: Re: OSD's slow down to a crawl
>>
>> Hi Matt,
>>
>> Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
>> Sebastien Han made a nice little blog entry describing it:
>>
>> http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
>>
>> You can get a lot of information about the current state of things.
>>
>> Mark
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
[not found] ` <38A500831D3DE24B90BD200D6C8701351BB15433@Exchange2010-2.corit.local>
@ 2012-12-20 17:13 ` Sage Weil
2012-12-21 4:51 ` Matthew Anderson
0 siblings, 1 reply; 13+ messages in thread
From: Sage Weil @ 2012-12-20 17:13 UTC (permalink / raw)
To: Matthew Anderson; +Cc: 'Mark Nelson', ceph-devel@vger.kernel.org
[-- Attachment #1: Type: TEXT/PLAIN, Size: 44581 bytes --]
On Thu, 20 Dec 2012, Matthew Anderson wrote:
> Hi Sage,
>
> Logs are attached. I took the osd logs from osd.24 as this is the first
> osd in my SSD pool I've been testing with previously.
>
> The 4MB bench I was able to reproduce the fault by restarting my rbd
> export which stalled after a few percent complete. When I ran the 4MB
> bench it stalled early on and never received a response back from the
> OSD and I terminated it after 60 seconds or so. I wasn't able to
> reproduce the fault using the 4kb io size. The 4kb log should show rados
> bench completing normally at a respectable speed of about 1MB/s.
Let's drill into the hang.. up until that point things look okay.
2012-12-21 00:51:26.033622 7f6f3c042760 1 -- 172.16.0.13:0/1023886 --> 172.16.0.13:6813/22233 -- osd_op(client.9503.0:185 benchmark_data_KVM04_23886_object184 [write 0~4194304] 6.3ca4346e) v4 -- ?+0 0x171ea50 con 0x171a7e0
Do you have a log for that OSD so we can see what happened there? It may
also be that the replicated write is hung. If you do
ceph pg map 6.3ca4346e
you can see all OSDs storing that PG. And/or you can grep for
client.9503.0:185 in 172.16.0.13:6813/22233's log and see whether the
sub_op was sent.
Thanks!
sage
>
> Thanks
> -Matt
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, 21 December 2012 12:30 AM
> To: Matthew Anderson
> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> Subject: RE: OSD's slow down to a crawl
>
> Can you do a similar test, but with full logging on?
>
> ceph tell osd.0 injectargs '--debug-ms 1 --debug-filestore 20 --debug-osd
> 20 --debug-journal 20'
> rados -p ssd bench 30 write -b 4096 -t 1 --log-file /tmp/foo --debug-ms 1
>
> That will be a single IO in flight at a time and very easy to trace through the logs. If you can post the resulting log file (/tmp/foo and from osd.0), that would be awesome.
>
> Thanks!
> sage
>
>
>
> On Thu, 20 Dec 2012, Matthew Anderson wrote:
>
> > # rados bench 60 write -t 256 -p ssd
> > Maintaining 256 concurrent writes of 4194304 bytes for at least 60 seconds.
> > Object prefix: benchmark_data_KVM03_12985
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 0 0 0 0 0 0 - 0
> > 1 45 45 0 0 0 - 0
> > 2 71 71 0 0 0 - 0
> > 3 90 90 0 0 0 - 0
> > 4 125 125 0 0 0 - 0
> > 5 136 136 0 0 0 - 0
> > 6 144 144 0 0 0 - 0
> > 7 175 175 0 0 0 - 0
> > 8 201 201 0 0 0 - 0
> > 9 217 217 0 0 0 - 0
> > 10 228 228 0 0 0 - 0
> > 11 251 251 0 0 0 - 0
> > 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> > 13 255 271 16 4.92194 20 12.9795 11.9979
> > 14 255 291 36 10.2833 80 13.1795 12.6478
> > 15 255 301 46 12.2639 40 13.9424 12.9123
> > 16 255 303 48 11.9973 8 14.0277 12.9579
> > 17 255 321 66 15.526 72 15.1408 13.4616
> > 18 255 345 90 19.9956 96 15.5673 13.9937
> > 19 255 349 94 19.7851 16 15.707 14.0635
> > 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 20 255 361 106 21.1953 48 16.4053 14.3121
> > 21 255 367 112 21.3285 24 17.4263 14.4795
> > 22 255 370 115 20.9043 12 17.3579 14.5544
> > 23 255 387 132 22.9513 68 18.3185 15.0867
> > 24 255 394 139 23.1614 28 18.2868 15.2504
> > 25 255 404 149 23.8346 40 18.741 15.5048
> > 26 255 410 155 23.8407 24 19.032 15.6358
> > 27 255 410 155 22.9577 0 - 15.6358
> > 28 255 410 155 22.1378 0 - 15.6358
> > 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> > 30 255 431 176 23.4614 80 22.6577 16.4822
> > 31 255 447 192 24.7687 64 23.2345 17.0563
> > 32 255 464 209 26.1192 68 23.3788 17.5859
> > 33 255 485 230 27.8726 84 22.8924 18.1122
> > 34 255 497 242 28.4642 48 23.563 18.3693
> > 35 255 508 253 28.9078 44 23.0516 18.5935
> > 36 255 514 259 28.7713 24 24.4415 18.7278
> > 37 255 530 275 29.723 64 23.847 19.0424
> > 38 255 541 286 30.0985 44 24.5757 19.2427
> > 39 255 544 289 29.6344 12 24.6802 19.2984
> > 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 40 255 544 289 28.8935 0 - 19.2984
> > 41 255 544 289 28.1888 0 - 19.2984
> > 42 255 544 289 27.5176 0 - 19.2984
> > 43 255 544 289 26.8776 0 - 19.2984
> > 44 255 544 289 26.2668 0 - 19.2984
> > 45 255 544 289 25.6831 0 - 19.2984
> > 46 255 544 289 25.1248 0 - 19.2984
> > 47 255 544 289 24.5902 0 - 19.2984
> > 48 255 544 289 24.0779 0 - 19.2984
> > 49 255 544 289 23.5865 0 - 19.2984
> > 50 255 544 289 23.1148 0 - 19.2984
> > 51 255 544 289 22.6616 0 - 19.2984
> > 52 255 544 289 22.2258 0 - 19.2984
> > 53 255 544 289 21.8065 0 - 19.2984
> > 54 255 544 289 21.4026 0 - 19.2984
> > 55 255 544 289 21.0135 0 - 19.2984
> > 56 255 544 289 20.6383 0 - 19.2984
> > 57 255 544 289 20.2762 0 - 19.2984
> > 58 255 544 289 19.9266 0 - 19.2984
> > 59 255 544 289 19.5889 0 - 19.2984
> > 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 60 255 544 289 19.2624 0 - 19.2984
> > 61 255 544 289 18.9466 0 - 19.2984
> > 62 255 544 289 18.641 0 - 19.2984
> > 63 255 544 289 18.3452 0 - 19.2984
> > 64 255 544 289 18.0585 0 - 19.2984
> > 65 255 544 289 17.7807 0 - 19.2984
> > 66 255 544 289 17.5113 0 - 19.2984
> > 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> > Total time run: 68.000740
> > Total writes made: 545
> > Write size: 4194304
> > Bandwidth (MB/sec): 32.058
> >
> > Stddev Bandwidth: 26.8837
> > Max bandwidth (MB/sec): 96
> > Min bandwidth (MB/sec): 0
> > Average Latency: 29.2801
> > Stddev Latency: 11.9768
> > Max latency: 52.6277
> > Min latency: 11.4904
> >
> >
> >
> >
> >
> > ----Original Message-----
> > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > Sent: Thursday, 20 December 2012 11:59 PM
> > To: Matthew Anderson
> > Cc: ceph-devel@vger.kernel.org
> > Subject: Re: OSD's slow down to a crawl
> >
> > Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
> >
> > Mark
> >
> > On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> > > Hi Mark,
> > >
> > > Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
> > >
> > > It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
> > >
> > > The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> > > # ceph --admin-daemon ./ceph-osd.24.asok perf dump
> > >
> > > {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"jou
> > > rn
> > > al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_byt
> > > es
> > > ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"
> > > su
> > > m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":
> > > 13
> > > 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":
> > > 20
> > > 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":171023
> > > 25
> > > 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
> > > :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":17
> > > 08
> > > .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.241850000
> > > },
> > > "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"av
> > > gc
> > > ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
> > > :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock":{
> > > "w
> > > ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_lo
> > > ck
> > > ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
> > > apply_lo
> > ck":{"wai
> > t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_loc
> > k":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManager
> > ::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op_w
> > ip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"op
> > _latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_out
> > _bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.833689000
> > },"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"
> > sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684000}
> > ,"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avgco
> > unt":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.000000
> > 000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgcoun
> > t":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"
> > subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"
> > subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,"s
> > ubop_push_
> > in_bytes"
> > :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"pus
> > h":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops":0
> > ,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"num
> > pg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_fro
> > m_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_epo
> > ch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"get
> > ":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":20
> > 868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"av
> > gcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":
> > 500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"
> > take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"avgc
> > ount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":
> > {"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fail_
> > fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"put
> > _sum":7955
> > 46573,"wa
> > it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttle
> > r-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,"g
> > et_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put"
> > :18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},"t
> > hrottle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"ge
> > t":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":0,
> > "take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount":0
> > ,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"val
> > ":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"
> > get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":1990
> > 92,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_bytes
> > ":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_fai
> > l_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,"p
> > ut_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
> > >
> > > # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
> > >
> > > { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> > > "received_at": "2012-12-20 23:34:42.012131",
> > > "age": "75.428051",
> > > "duration": "0.009988",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101498},
> > > "events": [
> > > { "time": "2012-12-20 23:34:42.012317",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:42.012425",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:42.012678",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:42.012720",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:42.012815",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:34:42.012958",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:34:42.013033",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:34:42.013745",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:34:42.013803",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:34:42.017414",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:34:42.022066",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:34:42.022119",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:34:42.017048",
> > > "age": "75.423134",
> > > "duration": "0.000411",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:34:42.017218",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:42.017363",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:42.017401",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:42.017459",
> > > "event": "done"}]},
> > > { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> > > "received_at": "2012-12-20 23:34:49.851021",
> > > "age": "67.589161",
> > > "duration": "0.007114",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101502},
> > > "events": [
> > > { "time": "2012-12-20 23:34:49.851276",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:49.851429",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:49.851616",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:49.851648",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:49.851712",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:34:49.851770",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:34:49.851915",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:34:49.853956",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:34:49.853992",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:34:49.857016",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:34:49.858077",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:34:49.858135",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:34:49.856762",
> > > "age": "67.583420",
> > > "duration": "0.000308",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:34:49.856869",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:49.856969",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:49.857001",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:49.857070",
> > > "event": "done"}]},
> > > { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> > > "received_at": "2012-12-20 23:35:34.351518",
> > > "age": "23.088664",
> > > "duration": "0.009430",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101504},
> > > "events": [
> > > { "time": "2012-12-20 23:35:34.351714",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:34.351811",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:34.352030",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:34.352071",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:34.352146",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:35:34.352213",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:35:34.352287",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:35:34.355390",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:35:34.355475",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:35:34.356661",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:35:34.360894",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:35:34.360948",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:35:34.356286",
> > > "age": "23.083896",
> > > "duration": "0.000421",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:35:34.356489",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:34.356598",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:34.356640",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:34.356707",
> > > "event": "done"}]},
> > > { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> > > "received_at": "2012-12-20 23:35:42.133103",
> > > "age": "15.307079",
> > > "duration": "0.008357",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101512},
> > > "events": [
> > > { "time": "2012-12-20 23:35:42.133315",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:42.133427",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:42.133684",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:42.133723",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:42.133796",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:35:42.133908",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:35:42.133995",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:35:42.136965",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:35:42.137025",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:35:42.138456",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:35:42.141408",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:35:42.141460",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:35:42.138087",
> > > "age": "15.302095",
> > > "duration": "0.000410",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:35:42.138248",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:42.138363",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:42.138443",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:42.138497",
> > > "event": "done"}]}]}
> > >
> > > # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
> > > "osd.24",
> > > "cluster": "ceph",
> > > "none": "0\/5",
> > > "lockdep": "0\/1",
> > > "context": "0\/1",
> > > "crush": "1\/1",
> > > "mds": "1\/5",
> > > "mds_balancer": "1\/5",
> > > "mds_locker": "1\/5",
> > > "mds_log": "1\/5",
> > > "mds_log_expire": "1\/5",
> > > "mds_migrator": "1\/5",
> > > "buffer": "0\/1",
> > > "timer": "0\/1",
> > > "filer": "0\/1",
> > > "striper": "0\/1",
> > > "objecter": "0\/1",
> > > "rados": "0\/5",
> > > "rbd": "0\/5",
> > > "journaler": "0\/5",
> > > "objectcacher": "0\/5",
> > > "client": "0\/5",
> > > "osd": "0\/5",
> > > "optracker": "0\/5",
> > > "objclass": "0\/5",
> > > "filestore": "1\/3",
> > > "journal": "1\/3",
> > > "ms": "0\/5",
> > > "mon": "1\/5",
> > > "monc": "0\/10",
> > > "paxos": "0\/5",
> > > "tp": "0\/5",
> > > "auth": "1\/5",
> > > "crypto": "1\/5",
> > > "finisher": "1\/1",
> > > "heartbeatmap": "1\/5",
> > > "perfcounter": "1\/5",
> > > "rgw": "1\/5",
> > > "hadoop": "1\/5",
> > > "javaclient": "1\/5",
> > > "asok": "1\/5",
> > > "throttle": "1\/1",
> > > "host": "KVM04",
> > > "fsid": "00000000-0000-0000-0000-000000000000",
> > > "public_addr": ":\/0",
> > > "cluster_addr": ":\/0",
> > > "public_network": "",
> > > "cluster_network": "",
> > > "num_client": "1",
> > > "monmap": "",
> > > "mon_host": "",
> > > "lockdep": "false",
> > > "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> > > "daemonize": "true",
> > > "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> > > "chdir": "\/",
> > > "max_open_files": "0",
> > > "fatal_signal_handlers": "true",
> > > "log_file": "\/var\/log\/ceph\/osd.24.log",
> > > "log_max_new": "1000",
> > > "log_max_recent": "100000",
> > > "log_to_stderr": "false",
> > > "err_to_stderr": "true",
> > > "log_to_syslog": "false",
> > > "err_to_syslog": "false",
> > > "log_flush_on_exit": "true",
> > > "clog_to_monitors": "true",
> > > "clog_to_syslog": "false",
> > > "mon_cluster_log_to_syslog": "false",
> > > "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> > > "key": "",
> > > "keyfile": "",
> > > "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> > > "heartbeat_interval": "5",
> > > "heartbeat_file": "",
> > > "perf": "true",
> > > "ms_tcp_nodelay": "true",
> > > "ms_initial_backoff": "0.2",
> > > "ms_max_backoff": "15",
> > > "ms_nocrc": "false",
> > > "ms_die_on_bad_msg": "false",
> > > "ms_dispatch_throttle_bytes": "104857600",
> > > "ms_bind_ipv6": "false",
> > > "ms_bind_port_min": "6800",
> > > "ms_bind_port_max": "7100",
> > > "ms_rwthread_stack_bytes": "1048576",
> > > "ms_tcp_read_timeout": "900",
> > > "ms_inject_socket_failures": "0",
> > > "ms_inject_delay_type": "",
> > > "ms_inject_delay_max": "1",
> > > "ms_inject_delay_probability": "0",
> > > "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> > > "mon_initial_members": "",
> > > "mon_sync_fs_threshold": "5",
> > > "mon_tick_interval": "5",
> > > "mon_subscribe_interval": "300",
> > > "mon_osd_laggy_halflife": "3600",
> > > "mon_osd_laggy_weight": "0.3",
> > > "mon_osd_adjust_heartbeat_grace": "true",
> > > "mon_osd_adjust_down_out_interval": "true",
> > > "mon_osd_auto_mark_in": "false",
> > > "mon_osd_auto_mark_auto_out_in": "true",
> > > "mon_osd_auto_mark_new_in": "true",
> > > "mon_osd_down_out_interval": "300",
> > > "mon_osd_min_up_ratio": "0.3",
> > > "mon_osd_min_in_ratio": "0.3",
> > > "mon_lease": "5",
> > > "mon_lease_renew_interval": "3",
> > > "mon_lease_ack_timeout": "10",
> > > "mon_clock_drift_allowed": "0.05",
> > > "mon_clock_drift_warn_backoff": "5",
> > > "mon_accept_timeout": "10",
> > > "mon_pg_create_interval": "30",
> > > "mon_pg_stuck_threshold": "300",
> > > "mon_osd_full_ratio": "0.95",
> > > "mon_osd_nearfull_ratio": "0.85",
> > > "mon_globalid_prealloc": "100",
> > > "mon_osd_report_timeout": "900",
> > > "mon_force_standby_active": "true",
> > > "mon_min_osdmap_epochs": "500",
> > > "mon_max_pgmap_epochs": "500",
> > > "mon_max_log_epochs": "500",
> > > "mon_max_osd": "10000",
> > > "mon_probe_timeout": "2",
> > > "mon_slurp_timeout": "10",
> > > "mon_slurp_bytes": "262144",
> > > "mon_client_bytes": "104857600",
> > > "mon_daemon_bytes": "419430400",
> > > "mon_max_log_entries_per_event": "4096",
> > > "paxos_max_join_drift": "10",
> > > "paxos_propose_interval": "1",
> > > "paxos_min_wait": "0.05",
> > > "clock_offset": "0",
> > > "auth_cluster_required": "none",
> > > "auth_service_required": "none",
> > > "auth_client_required": "none",
> > > "auth_supported": "none",
> > > "cephx_require_signatures": "false",
> > > "cephx_sign_messages": "true",
> > > "auth_mon_ticket_ttl": "43200",
> > > "auth_service_ticket_ttl": "3600",
> > > "mon_client_hunt_interval": "3",
> > > "mon_client_ping_interval": "10",
> > > "mon_client_max_log_entries_per_message": "1000",
> > > "mon_max_pool_pg_num": "65536",
> > > "client_cache_size": "16384",
> > > "client_cache_mid": "0.75",
> > > "client_use_random_mds": "false",
> > > "client_mount_timeout": "30",
> > > "client_tick_interval": "1",
> > > "client_trace": "",
> > > "client_readahead_min": "131072",
> > > "client_readahead_max_bytes": "0",
> > > "client_readahead_max_periods": "4",
> > > "client_snapdir": ".snap",
> > > "client_mountpoint": "\/",
> > > "client_notify_timeout": "10",
> > > "client_caps_release_delay": "5",
> > > "client_oc": "true",
> > > "client_oc_size": "209715200",
> > > "client_oc_max_dirty": "104857600",
> > > "client_oc_target_dirty": "8388608",
> > > "client_oc_max_dirty_age": "5",
> > > "client_oc_max_objects": "1000",
> > > "client_debug_force_sync_read": "false",
> > > "client_debug_inject_tick_delay": "0",
> > > "fuse_use_invalidate_cb": "false",
> > > "fuse_big_writes": "true",
> > > "fuse_debug": "false",
> > > "objecter_tick_interval": "5",
> > > "objecter_timeout": "10",
> > > "objecter_inflight_op_bytes": "104857600",
> > > "objecter_inflight_ops": "1024",
> > > "journaler_allow_split_entries": "true",
> > > "journaler_write_head_interval": "15",
> > > "journaler_prefetch_periods": "10",
> > > "journaler_prezero_periods": "5",
> > > "journaler_batch_interval": "0.001",
> > > "journaler_batch_max": "0",
> > > "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> > > "mds_max_file_size": "1099511627776",
> > > "mds_cache_size": "100000",
> > > "mds_cache_mid": "0.7",
> > > "mds_mem_max": "1048576",
> > > "mds_dir_commit_ratio": "0.5",
> > > "mds_dir_max_commit_size": "90",
> > > "mds_decay_halflife": "5",
> > > "mds_beacon_interval": "4",
> > > "mds_beacon_grace": "15",
> > > "mds_blacklist_interval": "1440",
> > > "mds_session_timeout": "60",
> > > "mds_session_autoclose": "300",
> > > "mds_reconnect_timeout": "45",
> > > "mds_tick_interval": "5",
> > > "mds_dirstat_min_interval": "1",
> > > "mds_scatter_nudge_interval": "5",
> > > "mds_client_prealloc_inos": "1000",
> > > "mds_early_reply": "true",
> > > "mds_use_tmap": "true",
> > > "mds_default_dir_hash": "2",
> > > "mds_log": "true",
> > > "mds_log_skip_corrupt_events": "false",
> > > "mds_log_max_events": "-1",
> > > "mds_log_max_segments": "30",
> > > "mds_log_max_expiring": "20",
> > > "mds_bal_sample_interval": "3",
> > > "mds_bal_replicate_threshold": "8000",
> > > "mds_bal_unreplicate_threshold": "0",
> > > "mds_bal_frag": "false",
> > > "mds_bal_split_size": "10000",
> > > "mds_bal_split_rd": "25000",
> > > "mds_bal_split_wr": "10000",
> > > "mds_bal_split_bits": "3",
> > > "mds_bal_merge_size": "50",
> > > "mds_bal_merge_rd": "1000",
> > > "mds_bal_merge_wr": "1000",
> > > "mds_bal_interval": "10",
> > > "mds_bal_fragment_interval": "5",
> > > "mds_bal_idle_threshold": "0",
> > > "mds_bal_max": "-1",
> > > "mds_bal_max_until": "-1",
> > > "mds_bal_mode": "0",
> > > "mds_bal_min_rebalance": "0.1",
> > > "mds_bal_min_start": "0.2",
> > > "mds_bal_need_min": "0.8",
> > > "mds_bal_need_max": "1.2",
> > > "mds_bal_midchunk": "0.3",
> > > "mds_bal_minchunk": "0.001",
> > > "mds_bal_target_removal_min": "5",
> > > "mds_bal_target_removal_max": "10",
> > > "mds_replay_interval": "1",
> > > "mds_shutdown_check": "0",
> > > "mds_thrash_exports": "0",
> > > "mds_thrash_fragments": "0",
> > > "mds_dump_cache_on_map": "false",
> > > "mds_dump_cache_after_rejoin": "false",
> > > "mds_verify_scatter": "false",
> > > "mds_debug_scatterstat": "false",
> > > "mds_debug_frag": "false",
> > > "mds_debug_auth_pins": "false",
> > > "mds_debug_subtrees": "false",
> > > "mds_kill_mdstable_at": "0",
> > > "mds_kill_export_at": "0",
> > > "mds_kill_import_at": "0",
> > > "mds_kill_link_at": "0",
> > > "mds_kill_rename_at": "0",
> > > "mds_wipe_sessions": "false",
> > > "mds_wipe_ino_prealloc": "false",
> > > "mds_skip_ino": "0",
> > > "max_mds": "1",
> > > "mds_standby_for_name": "",
> > > "mds_standby_for_rank": "-1",
> > > "mds_standby_replay": "false",
> > > "osd_auto_upgrade_tmap": "true",
> > > "osd_tmapput_sets_uses_tmap": "false",
> > > "osd_max_backfills": "10",
> > > "osd_backfill_full_ratio": "0.85",
> > > "osd_backfill_retry_interval": "10",
> > > "osd_uuid": "00000000-0000-0000-0000-000000000000",
> > > "osd_data": "\/srv\/ceph\/osd\/osd.24",
> > > "osd_journal": "\/dev\/sda3",
> > > "osd_journal_size": "5120",
> > > "osd_max_write_size": "90",
> > > "osd_max_pgls": "1024",
> > > "osd_client_message_size_cap": "524288000",
> > > "osd_pg_bits": "6",
> > > "osd_pgp_bits": "6",
> > > "osd_min_rep": "1",
> > > "osd_max_rep": "10",
> > > "osd_pool_default_crush_rule": "0",
> > > "osd_pool_default_size": "2",
> > > "osd_pool_default_min_size": "0",
> > > "osd_pool_default_pg_num": "8",
> > > "osd_pool_default_pgp_num": "8",
> > > "osd_map_dedup": "true",
> > > "osd_map_cache_size": "500",
> > > "osd_map_message_max": "100",
> > > "osd_op_threads": "2",
> > > "osd_disk_threads": "1",
> > > "osd_recovery_threads": "1",
> > > "osd_recover_clone_overlap": "true",
> > > "osd_backfill_scan_min": "64",
> > > "osd_backfill_scan_max": "512",
> > > "osd_op_thread_timeout": "30",
> > > "osd_recovery_thread_timeout": "30",
> > > "osd_snap_trim_thread_timeout": "3600",
> > > "osd_scrub_thread_timeout": "60",
> > > "osd_scrub_finalize_thread_timeout": "600",
> > > "osd_remove_thread_timeout": "3600",
> > > "osd_command_thread_timeout": "600",
> > > "osd_age": "0.8",
> > > "osd_age_time": "0",
> > > "osd_heartbeat_addr": ":\/0",
> > > "osd_heartbeat_interval": "6",
> > > "osd_heartbeat_grace": "20",
> > > "osd_mon_heartbeat_interval": "30",
> > > "osd_mon_report_interval_max": "120",
> > > "osd_mon_report_interval_min": "5",
> > > "osd_mon_ack_timeout": "30",
> > > "osd_min_down_reporters": "1",
> > > "osd_min_down_reports": "3",
> > > "osd_default_data_pool_replay_window": "45",
> > > "osd_preserve_trimmed_log": "false",
> > > "osd_auto_mark_unfound_lost": "false",
> > > "osd_recovery_delay_start": "0",
> > > "osd_recovery_max_active": "50",
> > > "osd_recovery_max_chunk": "8388608",
> > > "osd_recovery_forget_lost_objects": "false",
> > > "osd_max_scrubs": "1",
> > > "osd_scrub_load_threshold": "0.5",
> > > "osd_scrub_min_interval": "300",
> > > "osd_scrub_max_interval": "86400",
> > > "osd_deep_scrub_interval": "604800",
> > > "osd_deep_scrub_stride": "524288",
> > > "osd_auto_weight": "false",
> > > "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> > > "osd_check_for_log_corruption": "false",
> > > "osd_use_stale_snap": "false",
> > > "osd_rollback_to_cluster_snap": "",
> > > "osd_default_notify_timeout": "30",
> > > "osd_kill_backfill_at": "0",
> > > "osd_min_pg_log_entries": "1000",
> > > "osd_op_complaint_time": "30",
> > > "osd_command_max_records": "256",
> > > "osd_op_log_threshold": "5",
> > > "osd_verify_sparse_read_holes": "false",
> > > "osd_debug_drop_ping_probability": "0",
> > > "osd_debug_drop_ping_duration": "0",
> > > "osd_debug_drop_pg_create_probability": "0",
> > > "osd_debug_drop_pg_create_duration": "1",
> > > "osd_debug_drop_op_probability": "0",
> > > "osd_op_history_size": "20",
> > > "osd_op_history_duration": "600",
> > > "osd_target_transaction_size": "300",
> > > "osd_client_op_priority": "63",
> > > "osd_recovery_op_priority": "10",
> > > "filestore": "false",
> > > "filestore_index_retry_probability": "0",
> > > "filestore_debug_omap_check": "false",
> > > "filestore_xattr_use_omap": "false",
> > > "filestore_max_inline_xattr_size": "512",
> > > "filestore_max_inline_xattrs": "2",
> > > "filestore_max_sync_interval": "5",
> > > "filestore_min_sync_interval": "0.01",
> > > "filestore_btrfs_snap": "true",
> > > "filestore_btrfs_clone_range": "true",
> > > "filestore_fsync_flushes_journal_data": "false",
> > > "filestore_fiemap": "false",
> > > "filestore_flusher": "true",
> > > "filestore_flusher_max_fds": "512",
> > > "filestore_flush_min": "65536",
> > > "filestore_sync_flush": "false",
> > > "filestore_journal_parallel": "false",
> > > "filestore_journal_writeahead": "false",
> > > "filestore_journal_trailing": "false",
> > > "filestore_queue_max_ops": "500",
> > > "filestore_queue_max_bytes": "104857600",
> > > "filestore_queue_committing_max_ops": "500",
> > > "filestore_queue_committing_max_bytes": "104857600",
> > > "filestore_op_threads": "2",
> > > "filestore_op_thread_timeout": "60",
> > > "filestore_op_thread_suicide_timeout": "180",
> > > "filestore_commit_timeout": "600",
> > > "filestore_fiemap_threshold": "4096",
> > > "filestore_merge_threshold": "10",
> > > "filestore_split_multiple": "2",
> > > "filestore_update_to": "1000",
> > > "filestore_blackhole": "false",
> > > "filestore_dump_file": "",
> > > "filestore_kill_at": "0",
> > > "filestore_fail_eio": "true",
> > > "journal_dio": "true",
> > > "journal_aio": "false",
> > > "journal_block_align": "true",
> > > "journal_max_write_bytes": "10485760",
> > > "journal_max_write_entries": "100",
> > > "journal_queue_max_ops": "500",
> > > "journal_queue_max_bytes": "104857600",
> > > "journal_align_min_size": "65536",
> > > "journal_replay_from": "0",
> > > "journal_zero_on_create": "false",
> > > "rbd_cache": "false",
> > > "rbd_cache_size": "33554432",
> > > "rbd_cache_max_dirty": "25165824",
> > > "rbd_cache_target_dirty": "16777216",
> > > "rbd_cache_max_dirty_age": "1",
> > > "nss_db_path": "",
> > > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> > > "rgw_enable_apis": "s3, swift, swift_auth, admin",
> > > "rgw_cache_enabled": "true",
> > > "rgw_cache_lru_size": "10000",
> > > "rgw_socket_path": "",
> > > "rgw_dns_name": "",
> > > "rgw_swift_url": "",
> > > "rgw_swift_url_prefix": "swift",
> > > "rgw_swift_auth_url": "",
> > > "rgw_swift_auth_entry": "auth",
> > > "rgw_keystone_url": "",
> > > "rgw_keystone_admin_token": "",
> > > "rgw_keystone_accepted_roles": "Member, admin",
> > > "rgw_keystone_token_cache_size": "10000",
> > > "rgw_keystone_revocation_interval": "900",
> > > "rgw_admin_entry": "admin",
> > > "rgw_enforce_swift_acls": "true",
> > > "rgw_swift_token_expiration": "86400",
> > > "rgw_print_continue": "true",
> > > "rgw_remote_addr_param": "REMOTE_ADDR",
> > > "rgw_op_thread_timeout": "600",
> > > "rgw_op_thread_suicide_timeout": "0",
> > > "rgw_thread_pool_size": "100",
> > > "rgw_num_control_oids": "8",
> > > "rgw_cluster_root_pool": ".rgw.root",
> > > "rgw_log_nonexistent_bucket": "false",
> > > "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > > "rgw_log_object_name_utc": "false",
> > > "rgw_usage_max_shards": "32",
> > > "rgw_usage_max_user_shards": "1",
> > > "rgw_enable_ops_log": "true",
> > > "rgw_enable_usage_log": "true",
> > > "rgw_ops_log_rados": "true",
> > > "rgw_ops_log_socket_path": "",
> > > "rgw_ops_log_data_backlog": "5242880",
> > > "rgw_usage_log_flush_threshold": "1024",
> > > "rgw_usage_log_tick_interval": "30",
> > > "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> > > "rgw_intent_log_object_name_utc": "false",
> > > "rgw_init_timeout": "30",
> > > "rgw_mime_types_file": "\/etc\/mime.types",
> > > "rgw_gc_max_objs": "32",
> > > "rgw_gc_obj_min_wait": "7200",
> > > "rgw_gc_processor_max_time": "3600",
> > > "rgw_gc_processor_period": "3600",
> > > "rgw_s3_success_create_obj_status": "0",
> > > "rgw_resolve_cname": "false",
> > > "rgw_obj_stripe_size": "4194304",
> > > "rgw_extended_http_attrs": "",
> > > "mutex_perf_counter": "false",
> > > "internal_safe_to_start_threads": "true"}
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > > Sent: Thursday, 20 December 2012 11:28 PM
> > > To: Matthew Anderson
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: OSD's slow down to a crawl
> > >
> > > Hi Matt,
> > >
> > > Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> > > Sebastien Han made a nice little blog entry describing it:
> > >
> > > http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
> > >
> > > You can get a lot of information about the current state of things.
> > >
> > > Mark
> > >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-20 17:13 ` Sage Weil
@ 2012-12-21 4:51 ` Matthew Anderson
2012-12-21 16:02 ` Sage Weil
0 siblings, 1 reply; 13+ messages in thread
From: Matthew Anderson @ 2012-12-21 4:51 UTC (permalink / raw)
To: 'Sage Weil'; +Cc: 'Mark Nelson', ceph-devel@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 45581 bytes --]
Hi Sage,
I've tried to reproduce the error again with logging on every OSD and got the above. RADOS bench had stalled on a write request like the last time and the attached log is the grep'd OSD log (# cat osd.25.log | grep client.9501.0:744 > freeze.log) . The OSD that stalled was 25, pg map is below -
# ceph pg map 6.5d83495b
osdmap e3775 pg 6.5d83495b (6.95b) -> up [25,31] acting [25,31]
I hope that's what you were after, if not just let me know
Thanks again
-Matt
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Friday, 21 December 2012 1:14 AM
To: Matthew Anderson
Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
Subject: RE: OSD's slow down to a crawl
On Thu, 20 Dec 2012, Matthew Anderson wrote:
> Hi Sage,
>
> Logs are attached. I took the osd logs from osd.24 as this is the
> first osd in my SSD pool I've been testing with previously.
>
> The 4MB bench I was able to reproduce the fault by restarting my rbd
> export which stalled after a few percent complete. When I ran the 4MB
> bench it stalled early on and never received a response back from the
> OSD and I terminated it after 60 seconds or so. I wasn't able to
> reproduce the fault using the 4kb io size. The 4kb log should show
> rados bench completing normally at a respectable speed of about 1MB/s.
Let's drill into the hang.. up until that point things look okay.
2012-12-21 00:51:26.033622 7f6f3c042760 1 -- 172.16.0.13:0/1023886 --> 172.16.0.13:6813/22233 -- osd_op(client.9503.0:185 benchmark_data_KVM04_23886_object184 [write 0~4194304] 6.3ca4346e) v4 -- ?+0 0x171ea50 con 0x171a7e0
Do you have a log for that OSD so we can see what happened there? It may also be that the replicated write is hung. If you do
ceph pg map 6.3ca4346e
you can see all OSDs storing that PG. And/or you can grep for
client.9503.0:185 in 172.16.0.13:6813/22233's log and see whether the sub_op was sent.
Thanks!
sage
>
> Thanks
> -Matt
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, 21 December 2012 12:30 AM
> To: Matthew Anderson
> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> Subject: RE: OSD's slow down to a crawl
>
> Can you do a similar test, but with full logging on?
>
> ceph tell osd.0 injectargs '--debug-ms 1 --debug-filestore 20
> --debug-osd
> 20 --debug-journal 20'
> rados -p ssd bench 30 write -b 4096 -t 1 --log-file /tmp/foo
> --debug-ms 1
>
> That will be a single IO in flight at a time and very easy to trace through the logs. If you can post the resulting log file (/tmp/foo and from osd.0), that would be awesome.
>
> Thanks!
> sage
>
>
>
> On Thu, 20 Dec 2012, Matthew Anderson wrote:
>
> > # rados bench 60 write -t 256 -p ssd Maintaining 256 concurrent
> > writes of 4194304 bytes for at least 60 seconds.
> > Object prefix: benchmark_data_KVM03_12985
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 0 0 0 0 0 0 - 0
> > 1 45 45 0 0 0 - 0
> > 2 71 71 0 0 0 - 0
> > 3 90 90 0 0 0 - 0
> > 4 125 125 0 0 0 - 0
> > 5 136 136 0 0 0 - 0
> > 6 144 144 0 0 0 - 0
> > 7 175 175 0 0 0 - 0
> > 8 201 201 0 0 0 - 0
> > 9 217 217 0 0 0 - 0
> > 10 228 228 0 0 0 - 0
> > 11 251 251 0 0 0 - 0
> > 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> > 13 255 271 16 4.92194 20 12.9795 11.9979
> > 14 255 291 36 10.2833 80 13.1795 12.6478
> > 15 255 301 46 12.2639 40 13.9424 12.9123
> > 16 255 303 48 11.9973 8 14.0277 12.9579
> > 17 255 321 66 15.526 72 15.1408 13.4616
> > 18 255 345 90 19.9956 96 15.5673 13.9937
> > 19 255 349 94 19.7851 16 15.707 14.0635
> > 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 20 255 361 106 21.1953 48 16.4053 14.3121
> > 21 255 367 112 21.3285 24 17.4263 14.4795
> > 22 255 370 115 20.9043 12 17.3579 14.5544
> > 23 255 387 132 22.9513 68 18.3185 15.0867
> > 24 255 394 139 23.1614 28 18.2868 15.2504
> > 25 255 404 149 23.8346 40 18.741 15.5048
> > 26 255 410 155 23.8407 24 19.032 15.6358
> > 27 255 410 155 22.9577 0 - 15.6358
> > 28 255 410 155 22.1378 0 - 15.6358
> > 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> > 30 255 431 176 23.4614 80 22.6577 16.4822
> > 31 255 447 192 24.7687 64 23.2345 17.0563
> > 32 255 464 209 26.1192 68 23.3788 17.5859
> > 33 255 485 230 27.8726 84 22.8924 18.1122
> > 34 255 497 242 28.4642 48 23.563 18.3693
> > 35 255 508 253 28.9078 44 23.0516 18.5935
> > 36 255 514 259 28.7713 24 24.4415 18.7278
> > 37 255 530 275 29.723 64 23.847 19.0424
> > 38 255 541 286 30.0985 44 24.5757 19.2427
> > 39 255 544 289 29.6344 12 24.6802 19.2984
> > 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 40 255 544 289 28.8935 0 - 19.2984
> > 41 255 544 289 28.1888 0 - 19.2984
> > 42 255 544 289 27.5176 0 - 19.2984
> > 43 255 544 289 26.8776 0 - 19.2984
> > 44 255 544 289 26.2668 0 - 19.2984
> > 45 255 544 289 25.6831 0 - 19.2984
> > 46 255 544 289 25.1248 0 - 19.2984
> > 47 255 544 289 24.5902 0 - 19.2984
> > 48 255 544 289 24.0779 0 - 19.2984
> > 49 255 544 289 23.5865 0 - 19.2984
> > 50 255 544 289 23.1148 0 - 19.2984
> > 51 255 544 289 22.6616 0 - 19.2984
> > 52 255 544 289 22.2258 0 - 19.2984
> > 53 255 544 289 21.8065 0 - 19.2984
> > 54 255 544 289 21.4026 0 - 19.2984
> > 55 255 544 289 21.0135 0 - 19.2984
> > 56 255 544 289 20.6383 0 - 19.2984
> > 57 255 544 289 20.2762 0 - 19.2984
> > 58 255 544 289 19.9266 0 - 19.2984
> > 59 255 544 289 19.5889 0 - 19.2984
> > 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > 60 255 544 289 19.2624 0 - 19.2984
> > 61 255 544 289 18.9466 0 - 19.2984
> > 62 255 544 289 18.641 0 - 19.2984
> > 63 255 544 289 18.3452 0 - 19.2984
> > 64 255 544 289 18.0585 0 - 19.2984
> > 65 255 544 289 17.7807 0 - 19.2984
> > 66 255 544 289 17.5113 0 - 19.2984
> > 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> > Total time run: 68.000740
> > Total writes made: 545
> > Write size: 4194304
> > Bandwidth (MB/sec): 32.058
> >
> > Stddev Bandwidth: 26.8837
> > Max bandwidth (MB/sec): 96
> > Min bandwidth (MB/sec): 0
> > Average Latency: 29.2801
> > Stddev Latency: 11.9768
> > Max latency: 52.6277
> > Min latency: 11.4904
> >
> >
> >
> >
> >
> > ----Original Message-----
> > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > Sent: Thursday, 20 December 2012 11:59 PM
> > To: Matthew Anderson
> > Cc: ceph-devel@vger.kernel.org
> > Subject: Re: OSD's slow down to a crawl
> >
> > Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
> >
> > Mark
> >
> > On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> > > Hi Mark,
> > >
> > > Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
> > >
> > > It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
> > >
> > > The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> > > # ceph --admin-daemon ./ceph-osd.24.asok perf dump
> > >
> > > {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"j
> > > ou
> > > rn
> > > al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_b
> > > yt
> > > es
> > > ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"
> > > su
> > > m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":
> > > 13
> > > 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":
> > > 20
> > > 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1710
> > > 23
> > > 25
> > > 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
> > > :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":
> > > 17
> > > 08
> > > .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.2418500
> > > 00
> > > },
> > > "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"
> > > av
> > > gc
> > > ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
> > > :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock"
> > > :{
> > > "w
> > > ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_
> > > lo
> > > ck
> > > ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
> > > apply_lo
> > ck":{"wai
> > t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_l
> > oc
> > k":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManag
> > er
> > ::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op
> > _w
> > ip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"
> > op
> > _latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_o
> > ut
> > _bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.8336890
> > 00
> > },"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"
> > sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.34968400
> > 0}
> > ,"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avg
> > co
> > unt":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.0000
> > 00
> > 000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgco
> > un
> > t":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"
> > subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"
> > subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,
> > "s
> > ubop_push_
> > in_bytes"
> > :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"p
> > us
> > h":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops"
> > :0
> > ,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"n
> > um
> > pg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_f
> > ro
> > m_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_e
> > po
> > ch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"g
> > et
> > ":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":
> > 20
> > 868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"
> > av
> > gcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":
> > 500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"
> > take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"av
> > gc
> > ount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":
> > {"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fai
> > l_
> > fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"p
> > ut
> > _sum":7955
> > 46573,"wa
> > it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_thrott
> > le
> > r-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,
> > "g
> > et_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put"
> > :18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},
> > "t
> > hrottle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"
> > ge
> > t":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":
> > 0,
> > "take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount"
> > :0
> > ,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"v
> > al
> > ":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"
> > get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":19
> > 90
> > 92,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_byt
> > es
> > ":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_f
> > ai
> > l_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,
> > "p ut_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
> > >
> > > # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
> > >
> > > { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> > > "received_at": "2012-12-20 23:34:42.012131",
> > > "age": "75.428051",
> > > "duration": "0.009988",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101498},
> > > "events": [
> > > { "time": "2012-12-20 23:34:42.012317",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:42.012425",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:42.012678",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:42.012720",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:42.012815",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:34:42.012958",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:34:42.013033",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:34:42.013745",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:34:42.013803",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:34:42.017414",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:34:42.022066",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:34:42.022119",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:34:42.017048",
> > > "age": "75.423134",
> > > "duration": "0.000411",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:34:42.017218",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:42.017363",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:42.017401",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:42.017459",
> > > "event": "done"}]},
> > > { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> > > "received_at": "2012-12-20 23:34:49.851021",
> > > "age": "67.589161",
> > > "duration": "0.007114",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101502},
> > > "events": [
> > > { "time": "2012-12-20 23:34:49.851276",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:49.851429",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:49.851616",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:49.851648",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:49.851712",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:34:49.851770",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:34:49.851915",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:34:49.853956",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:34:49.853992",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:34:49.857016",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:34:49.858077",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:34:49.858135",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:34:49.856762",
> > > "age": "67.583420",
> > > "duration": "0.000308",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:34:49.856869",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:34:49.856969",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:34:49.857001",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:34:49.857070",
> > > "event": "done"}]},
> > > { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> > > "received_at": "2012-12-20 23:35:34.351518",
> > > "age": "23.088664",
> > > "duration": "0.009430",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101504},
> > > "events": [
> > > { "time": "2012-12-20 23:35:34.351714",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:34.351811",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:34.352030",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:34.352071",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:34.352146",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:35:34.352213",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:35:34.352287",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:35:34.355390",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:35:34.355475",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:35:34.356661",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:35:34.360894",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:35:34.360948",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:35:34.356286",
> > > "age": "23.083896",
> > > "duration": "0.000421",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:35:34.356489",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:34.356598",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:34.356640",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:34.356707",
> > > "event": "done"}]},
> > > { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> > > "received_at": "2012-12-20 23:35:42.133103",
> > > "age": "15.307079",
> > > "duration": "0.008357",
> > > "flag_point": "waiting for sub ops",
> > > "client_info": { "client": "client.7636",
> > > "tid": 101512},
> > > "events": [
> > > { "time": "2012-12-20 23:35:42.133315",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:42.133427",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:42.133684",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:42.133723",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:42.133796",
> > > "event": "sub_op_sent"},
> > > { "time": "2012-12-20 23:35:42.133908",
> > > "event": "commit_queued_for_journal_write"},
> > > { "time": "2012-12-20 23:35:42.133995",
> > > "event": "write_thread_in_journal_buffer"},
> > > { "time": "2012-12-20 23:35:42.136965",
> > > "event": "journaled_completion_queued"},
> > > { "time": "2012-12-20 23:35:42.137025",
> > > "event": "op_commit"},
> > > { "time": "2012-12-20 23:35:42.138456",
> > > "event": "sub_op_commit_rec"},
> > > { "time": "2012-12-20 23:35:42.141408",
> > > "event": "op_applied"},
> > > { "time": "2012-12-20 23:35:42.141460",
> > > "event": "done"}]},
> > > { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > "received_at": "2012-12-20 23:35:42.138087",
> > > "age": "15.302095",
> > > "duration": "0.000410",
> > > "flag_point": "started",
> > > "events": [
> > > { "time": "2012-12-20 23:35:42.138248",
> > > "event": "waiting_for_osdmap"},
> > > { "time": "2012-12-20 23:35:42.138363",
> > > "event": "reached_pg"},
> > > { "time": "2012-12-20 23:35:42.138443",
> > > "event": "started"},
> > > { "time": "2012-12-20 23:35:42.138497",
> > > "event": "done"}]}]}
> > >
> > > # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
> > > "osd.24",
> > > "cluster": "ceph",
> > > "none": "0\/5",
> > > "lockdep": "0\/1",
> > > "context": "0\/1",
> > > "crush": "1\/1",
> > > "mds": "1\/5",
> > > "mds_balancer": "1\/5",
> > > "mds_locker": "1\/5",
> > > "mds_log": "1\/5",
> > > "mds_log_expire": "1\/5",
> > > "mds_migrator": "1\/5",
> > > "buffer": "0\/1",
> > > "timer": "0\/1",
> > > "filer": "0\/1",
> > > "striper": "0\/1",
> > > "objecter": "0\/1",
> > > "rados": "0\/5",
> > > "rbd": "0\/5",
> > > "journaler": "0\/5",
> > > "objectcacher": "0\/5",
> > > "client": "0\/5",
> > > "osd": "0\/5",
> > > "optracker": "0\/5",
> > > "objclass": "0\/5",
> > > "filestore": "1\/3",
> > > "journal": "1\/3",
> > > "ms": "0\/5",
> > > "mon": "1\/5",
> > > "monc": "0\/10",
> > > "paxos": "0\/5",
> > > "tp": "0\/5",
> > > "auth": "1\/5",
> > > "crypto": "1\/5",
> > > "finisher": "1\/1",
> > > "heartbeatmap": "1\/5",
> > > "perfcounter": "1\/5",
> > > "rgw": "1\/5",
> > > "hadoop": "1\/5",
> > > "javaclient": "1\/5",
> > > "asok": "1\/5",
> > > "throttle": "1\/1",
> > > "host": "KVM04",
> > > "fsid": "00000000-0000-0000-0000-000000000000",
> > > "public_addr": ":\/0",
> > > "cluster_addr": ":\/0",
> > > "public_network": "",
> > > "cluster_network": "",
> > > "num_client": "1",
> > > "monmap": "",
> > > "mon_host": "",
> > > "lockdep": "false",
> > > "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> > > "daemonize": "true",
> > > "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> > > "chdir": "\/",
> > > "max_open_files": "0",
> > > "fatal_signal_handlers": "true",
> > > "log_file": "\/var\/log\/ceph\/osd.24.log",
> > > "log_max_new": "1000",
> > > "log_max_recent": "100000",
> > > "log_to_stderr": "false",
> > > "err_to_stderr": "true",
> > > "log_to_syslog": "false",
> > > "err_to_syslog": "false",
> > > "log_flush_on_exit": "true",
> > > "clog_to_monitors": "true",
> > > "clog_to_syslog": "false",
> > > "mon_cluster_log_to_syslog": "false",
> > > "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> > > "key": "",
> > > "keyfile": "",
> > > "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> > > "heartbeat_interval": "5",
> > > "heartbeat_file": "",
> > > "perf": "true",
> > > "ms_tcp_nodelay": "true",
> > > "ms_initial_backoff": "0.2",
> > > "ms_max_backoff": "15",
> > > "ms_nocrc": "false",
> > > "ms_die_on_bad_msg": "false",
> > > "ms_dispatch_throttle_bytes": "104857600",
> > > "ms_bind_ipv6": "false",
> > > "ms_bind_port_min": "6800",
> > > "ms_bind_port_max": "7100",
> > > "ms_rwthread_stack_bytes": "1048576",
> > > "ms_tcp_read_timeout": "900",
> > > "ms_inject_socket_failures": "0",
> > > "ms_inject_delay_type": "",
> > > "ms_inject_delay_max": "1",
> > > "ms_inject_delay_probability": "0",
> > > "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> > > "mon_initial_members": "",
> > > "mon_sync_fs_threshold": "5",
> > > "mon_tick_interval": "5",
> > > "mon_subscribe_interval": "300",
> > > "mon_osd_laggy_halflife": "3600",
> > > "mon_osd_laggy_weight": "0.3",
> > > "mon_osd_adjust_heartbeat_grace": "true",
> > > "mon_osd_adjust_down_out_interval": "true",
> > > "mon_osd_auto_mark_in": "false",
> > > "mon_osd_auto_mark_auto_out_in": "true",
> > > "mon_osd_auto_mark_new_in": "true",
> > > "mon_osd_down_out_interval": "300",
> > > "mon_osd_min_up_ratio": "0.3",
> > > "mon_osd_min_in_ratio": "0.3",
> > > "mon_lease": "5",
> > > "mon_lease_renew_interval": "3",
> > > "mon_lease_ack_timeout": "10",
> > > "mon_clock_drift_allowed": "0.05",
> > > "mon_clock_drift_warn_backoff": "5",
> > > "mon_accept_timeout": "10",
> > > "mon_pg_create_interval": "30",
> > > "mon_pg_stuck_threshold": "300",
> > > "mon_osd_full_ratio": "0.95",
> > > "mon_osd_nearfull_ratio": "0.85",
> > > "mon_globalid_prealloc": "100",
> > > "mon_osd_report_timeout": "900",
> > > "mon_force_standby_active": "true",
> > > "mon_min_osdmap_epochs": "500",
> > > "mon_max_pgmap_epochs": "500",
> > > "mon_max_log_epochs": "500",
> > > "mon_max_osd": "10000",
> > > "mon_probe_timeout": "2",
> > > "mon_slurp_timeout": "10",
> > > "mon_slurp_bytes": "262144",
> > > "mon_client_bytes": "104857600",
> > > "mon_daemon_bytes": "419430400",
> > > "mon_max_log_entries_per_event": "4096",
> > > "paxos_max_join_drift": "10",
> > > "paxos_propose_interval": "1",
> > > "paxos_min_wait": "0.05",
> > > "clock_offset": "0",
> > > "auth_cluster_required": "none",
> > > "auth_service_required": "none",
> > > "auth_client_required": "none",
> > > "auth_supported": "none",
> > > "cephx_require_signatures": "false",
> > > "cephx_sign_messages": "true",
> > > "auth_mon_ticket_ttl": "43200",
> > > "auth_service_ticket_ttl": "3600",
> > > "mon_client_hunt_interval": "3",
> > > "mon_client_ping_interval": "10",
> > > "mon_client_max_log_entries_per_message": "1000",
> > > "mon_max_pool_pg_num": "65536",
> > > "client_cache_size": "16384",
> > > "client_cache_mid": "0.75",
> > > "client_use_random_mds": "false",
> > > "client_mount_timeout": "30",
> > > "client_tick_interval": "1",
> > > "client_trace": "",
> > > "client_readahead_min": "131072",
> > > "client_readahead_max_bytes": "0",
> > > "client_readahead_max_periods": "4",
> > > "client_snapdir": ".snap",
> > > "client_mountpoint": "\/",
> > > "client_notify_timeout": "10",
> > > "client_caps_release_delay": "5",
> > > "client_oc": "true",
> > > "client_oc_size": "209715200",
> > > "client_oc_max_dirty": "104857600",
> > > "client_oc_target_dirty": "8388608",
> > > "client_oc_max_dirty_age": "5",
> > > "client_oc_max_objects": "1000",
> > > "client_debug_force_sync_read": "false",
> > > "client_debug_inject_tick_delay": "0",
> > > "fuse_use_invalidate_cb": "false",
> > > "fuse_big_writes": "true",
> > > "fuse_debug": "false",
> > > "objecter_tick_interval": "5",
> > > "objecter_timeout": "10",
> > > "objecter_inflight_op_bytes": "104857600",
> > > "objecter_inflight_ops": "1024",
> > > "journaler_allow_split_entries": "true",
> > > "journaler_write_head_interval": "15",
> > > "journaler_prefetch_periods": "10",
> > > "journaler_prezero_periods": "5",
> > > "journaler_batch_interval": "0.001",
> > > "journaler_batch_max": "0",
> > > "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> > > "mds_max_file_size": "1099511627776",
> > > "mds_cache_size": "100000",
> > > "mds_cache_mid": "0.7",
> > > "mds_mem_max": "1048576",
> > > "mds_dir_commit_ratio": "0.5",
> > > "mds_dir_max_commit_size": "90",
> > > "mds_decay_halflife": "5",
> > > "mds_beacon_interval": "4",
> > > "mds_beacon_grace": "15",
> > > "mds_blacklist_interval": "1440",
> > > "mds_session_timeout": "60",
> > > "mds_session_autoclose": "300",
> > > "mds_reconnect_timeout": "45",
> > > "mds_tick_interval": "5",
> > > "mds_dirstat_min_interval": "1",
> > > "mds_scatter_nudge_interval": "5",
> > > "mds_client_prealloc_inos": "1000",
> > > "mds_early_reply": "true",
> > > "mds_use_tmap": "true",
> > > "mds_default_dir_hash": "2",
> > > "mds_log": "true",
> > > "mds_log_skip_corrupt_events": "false",
> > > "mds_log_max_events": "-1",
> > > "mds_log_max_segments": "30",
> > > "mds_log_max_expiring": "20",
> > > "mds_bal_sample_interval": "3",
> > > "mds_bal_replicate_threshold": "8000",
> > > "mds_bal_unreplicate_threshold": "0",
> > > "mds_bal_frag": "false",
> > > "mds_bal_split_size": "10000",
> > > "mds_bal_split_rd": "25000",
> > > "mds_bal_split_wr": "10000",
> > > "mds_bal_split_bits": "3",
> > > "mds_bal_merge_size": "50",
> > > "mds_bal_merge_rd": "1000",
> > > "mds_bal_merge_wr": "1000",
> > > "mds_bal_interval": "10",
> > > "mds_bal_fragment_interval": "5",
> > > "mds_bal_idle_threshold": "0",
> > > "mds_bal_max": "-1",
> > > "mds_bal_max_until": "-1",
> > > "mds_bal_mode": "0",
> > > "mds_bal_min_rebalance": "0.1",
> > > "mds_bal_min_start": "0.2",
> > > "mds_bal_need_min": "0.8",
> > > "mds_bal_need_max": "1.2",
> > > "mds_bal_midchunk": "0.3",
> > > "mds_bal_minchunk": "0.001",
> > > "mds_bal_target_removal_min": "5",
> > > "mds_bal_target_removal_max": "10",
> > > "mds_replay_interval": "1",
> > > "mds_shutdown_check": "0",
> > > "mds_thrash_exports": "0",
> > > "mds_thrash_fragments": "0",
> > > "mds_dump_cache_on_map": "false",
> > > "mds_dump_cache_after_rejoin": "false",
> > > "mds_verify_scatter": "false",
> > > "mds_debug_scatterstat": "false",
> > > "mds_debug_frag": "false",
> > > "mds_debug_auth_pins": "false",
> > > "mds_debug_subtrees": "false",
> > > "mds_kill_mdstable_at": "0",
> > > "mds_kill_export_at": "0",
> > > "mds_kill_import_at": "0",
> > > "mds_kill_link_at": "0",
> > > "mds_kill_rename_at": "0",
> > > "mds_wipe_sessions": "false",
> > > "mds_wipe_ino_prealloc": "false",
> > > "mds_skip_ino": "0",
> > > "max_mds": "1",
> > > "mds_standby_for_name": "",
> > > "mds_standby_for_rank": "-1",
> > > "mds_standby_replay": "false",
> > > "osd_auto_upgrade_tmap": "true",
> > > "osd_tmapput_sets_uses_tmap": "false",
> > > "osd_max_backfills": "10",
> > > "osd_backfill_full_ratio": "0.85",
> > > "osd_backfill_retry_interval": "10",
> > > "osd_uuid": "00000000-0000-0000-0000-000000000000",
> > > "osd_data": "\/srv\/ceph\/osd\/osd.24",
> > > "osd_journal": "\/dev\/sda3",
> > > "osd_journal_size": "5120",
> > > "osd_max_write_size": "90",
> > > "osd_max_pgls": "1024",
> > > "osd_client_message_size_cap": "524288000",
> > > "osd_pg_bits": "6",
> > > "osd_pgp_bits": "6",
> > > "osd_min_rep": "1",
> > > "osd_max_rep": "10",
> > > "osd_pool_default_crush_rule": "0",
> > > "osd_pool_default_size": "2",
> > > "osd_pool_default_min_size": "0",
> > > "osd_pool_default_pg_num": "8",
> > > "osd_pool_default_pgp_num": "8",
> > > "osd_map_dedup": "true",
> > > "osd_map_cache_size": "500",
> > > "osd_map_message_max": "100",
> > > "osd_op_threads": "2",
> > > "osd_disk_threads": "1",
> > > "osd_recovery_threads": "1",
> > > "osd_recover_clone_overlap": "true",
> > > "osd_backfill_scan_min": "64",
> > > "osd_backfill_scan_max": "512",
> > > "osd_op_thread_timeout": "30",
> > > "osd_recovery_thread_timeout": "30",
> > > "osd_snap_trim_thread_timeout": "3600",
> > > "osd_scrub_thread_timeout": "60",
> > > "osd_scrub_finalize_thread_timeout": "600",
> > > "osd_remove_thread_timeout": "3600",
> > > "osd_command_thread_timeout": "600",
> > > "osd_age": "0.8",
> > > "osd_age_time": "0",
> > > "osd_heartbeat_addr": ":\/0",
> > > "osd_heartbeat_interval": "6",
> > > "osd_heartbeat_grace": "20",
> > > "osd_mon_heartbeat_interval": "30",
> > > "osd_mon_report_interval_max": "120",
> > > "osd_mon_report_interval_min": "5",
> > > "osd_mon_ack_timeout": "30",
> > > "osd_min_down_reporters": "1",
> > > "osd_min_down_reports": "3",
> > > "osd_default_data_pool_replay_window": "45",
> > > "osd_preserve_trimmed_log": "false",
> > > "osd_auto_mark_unfound_lost": "false",
> > > "osd_recovery_delay_start": "0",
> > > "osd_recovery_max_active": "50",
> > > "osd_recovery_max_chunk": "8388608",
> > > "osd_recovery_forget_lost_objects": "false",
> > > "osd_max_scrubs": "1",
> > > "osd_scrub_load_threshold": "0.5",
> > > "osd_scrub_min_interval": "300",
> > > "osd_scrub_max_interval": "86400",
> > > "osd_deep_scrub_interval": "604800",
> > > "osd_deep_scrub_stride": "524288",
> > > "osd_auto_weight": "false",
> > > "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> > > "osd_check_for_log_corruption": "false",
> > > "osd_use_stale_snap": "false",
> > > "osd_rollback_to_cluster_snap": "",
> > > "osd_default_notify_timeout": "30",
> > > "osd_kill_backfill_at": "0",
> > > "osd_min_pg_log_entries": "1000",
> > > "osd_op_complaint_time": "30",
> > > "osd_command_max_records": "256",
> > > "osd_op_log_threshold": "5",
> > > "osd_verify_sparse_read_holes": "false",
> > > "osd_debug_drop_ping_probability": "0",
> > > "osd_debug_drop_ping_duration": "0",
> > > "osd_debug_drop_pg_create_probability": "0",
> > > "osd_debug_drop_pg_create_duration": "1",
> > > "osd_debug_drop_op_probability": "0",
> > > "osd_op_history_size": "20",
> > > "osd_op_history_duration": "600",
> > > "osd_target_transaction_size": "300",
> > > "osd_client_op_priority": "63",
> > > "osd_recovery_op_priority": "10",
> > > "filestore": "false",
> > > "filestore_index_retry_probability": "0",
> > > "filestore_debug_omap_check": "false",
> > > "filestore_xattr_use_omap": "false",
> > > "filestore_max_inline_xattr_size": "512",
> > > "filestore_max_inline_xattrs": "2",
> > > "filestore_max_sync_interval": "5",
> > > "filestore_min_sync_interval": "0.01",
> > > "filestore_btrfs_snap": "true",
> > > "filestore_btrfs_clone_range": "true",
> > > "filestore_fsync_flushes_journal_data": "false",
> > > "filestore_fiemap": "false",
> > > "filestore_flusher": "true",
> > > "filestore_flusher_max_fds": "512",
> > > "filestore_flush_min": "65536",
> > > "filestore_sync_flush": "false",
> > > "filestore_journal_parallel": "false",
> > > "filestore_journal_writeahead": "false",
> > > "filestore_journal_trailing": "false",
> > > "filestore_queue_max_ops": "500",
> > > "filestore_queue_max_bytes": "104857600",
> > > "filestore_queue_committing_max_ops": "500",
> > > "filestore_queue_committing_max_bytes": "104857600",
> > > "filestore_op_threads": "2",
> > > "filestore_op_thread_timeout": "60",
> > > "filestore_op_thread_suicide_timeout": "180",
> > > "filestore_commit_timeout": "600",
> > > "filestore_fiemap_threshold": "4096",
> > > "filestore_merge_threshold": "10",
> > > "filestore_split_multiple": "2",
> > > "filestore_update_to": "1000",
> > > "filestore_blackhole": "false",
> > > "filestore_dump_file": "",
> > > "filestore_kill_at": "0",
> > > "filestore_fail_eio": "true",
> > > "journal_dio": "true",
> > > "journal_aio": "false",
> > > "journal_block_align": "true",
> > > "journal_max_write_bytes": "10485760",
> > > "journal_max_write_entries": "100",
> > > "journal_queue_max_ops": "500",
> > > "journal_queue_max_bytes": "104857600",
> > > "journal_align_min_size": "65536",
> > > "journal_replay_from": "0",
> > > "journal_zero_on_create": "false",
> > > "rbd_cache": "false",
> > > "rbd_cache_size": "33554432",
> > > "rbd_cache_max_dirty": "25165824",
> > > "rbd_cache_target_dirty": "16777216",
> > > "rbd_cache_max_dirty_age": "1",
> > > "nss_db_path": "",
> > > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> > > "rgw_enable_apis": "s3, swift, swift_auth, admin",
> > > "rgw_cache_enabled": "true",
> > > "rgw_cache_lru_size": "10000",
> > > "rgw_socket_path": "",
> > > "rgw_dns_name": "",
> > > "rgw_swift_url": "",
> > > "rgw_swift_url_prefix": "swift",
> > > "rgw_swift_auth_url": "",
> > > "rgw_swift_auth_entry": "auth",
> > > "rgw_keystone_url": "",
> > > "rgw_keystone_admin_token": "",
> > > "rgw_keystone_accepted_roles": "Member, admin",
> > > "rgw_keystone_token_cache_size": "10000",
> > > "rgw_keystone_revocation_interval": "900",
> > > "rgw_admin_entry": "admin",
> > > "rgw_enforce_swift_acls": "true",
> > > "rgw_swift_token_expiration": "86400",
> > > "rgw_print_continue": "true",
> > > "rgw_remote_addr_param": "REMOTE_ADDR",
> > > "rgw_op_thread_timeout": "600",
> > > "rgw_op_thread_suicide_timeout": "0",
> > > "rgw_thread_pool_size": "100",
> > > "rgw_num_control_oids": "8",
> > > "rgw_cluster_root_pool": ".rgw.root",
> > > "rgw_log_nonexistent_bucket": "false",
> > > "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > > "rgw_log_object_name_utc": "false",
> > > "rgw_usage_max_shards": "32",
> > > "rgw_usage_max_user_shards": "1",
> > > "rgw_enable_ops_log": "true",
> > > "rgw_enable_usage_log": "true",
> > > "rgw_ops_log_rados": "true",
> > > "rgw_ops_log_socket_path": "",
> > > "rgw_ops_log_data_backlog": "5242880",
> > > "rgw_usage_log_flush_threshold": "1024",
> > > "rgw_usage_log_tick_interval": "30",
> > > "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> > > "rgw_intent_log_object_name_utc": "false",
> > > "rgw_init_timeout": "30",
> > > "rgw_mime_types_file": "\/etc\/mime.types",
> > > "rgw_gc_max_objs": "32",
> > > "rgw_gc_obj_min_wait": "7200",
> > > "rgw_gc_processor_max_time": "3600",
> > > "rgw_gc_processor_period": "3600",
> > > "rgw_s3_success_create_obj_status": "0",
> > > "rgw_resolve_cname": "false",
> > > "rgw_obj_stripe_size": "4194304",
> > > "rgw_extended_http_attrs": "",
> > > "mutex_perf_counter": "false",
> > > "internal_safe_to_start_threads": "true"}
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > > Sent: Thursday, 20 December 2012 11:28 PM
> > > To: Matthew Anderson
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: OSD's slow down to a crawl
> > >
> > > Hi Matt,
> > >
> > > Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> > > Sebastien Han made a nice little blog entry describing it:
> > >
> > > http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
> > >
> > > You can get a lot of information about the current state of things.
> > >
> > > Mark
> > >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
[-- Attachment #2: freeze.log --]
[-- Type: application/octet-stream, Size: 11078 bytes --]
2012-12-21 12:40:00.770821 7f4d90ff9700 1 -- 172.16.0.13:6814/15146 <== client.9501 172.16.0.13:0/1016365 118 ==== osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 ==== 175+0+4194304 (3920014038 0 2072446764) 0x7f4ca6c0d750 con 0x7f4cf40015c0
2012-12-21 12:40:00.770876 7f4d90ff9700 20 osd.25 3775 _dispatch 0x7f4ca6c0d750 osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4
2012-12-21 12:40:00.770980 7f4d90ff9700 15 osd.25 3775 enqueue_op 0x7f4d5c1f6720 osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4
2012-12-21 12:40:00.771094 7f4d87ff7700 10 osd.25 3775 dequeue_op 0x7f4d5c1f6720 osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 pg pg[6.95b( v 3775'134 (0'0,3775'134] local-les=3774 n=14 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean]
2012-12-21 12:40:00.771173 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'134 (0'0,3775'134] local-les=3774 n=14 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] do_op osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 may_write
2012-12-21 12:40:00.771874 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'134 (0'0,3775'134] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] append_log log(0'0,3775'134] [3775'135 (0'0) modify 5d83495b/benchmark_data_KVM04_16365_object743/head//6 by client.9501.0:744 2012-12-21 12:40:00.759292]
2012-12-21 12:40:00.771908 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] add_log_entry 3775'135 (0'0) modify 5d83495b/benchmark_data_KVM04_16365_object743/head//6 by client.9501.0:744 2012-12-21 12:40:00.759292
2012-12-21 12:40:00.772002 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] new_repop rep_tid 13299 on osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4
2012-12-21 12:40:00.772100 7f4d87ff7700 1 -- 172.16.0.13:6816/15146 --> osd.31 172.16.0.12:6822/17200 -- osd_sub_op(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] v 3775'135 snapset=0=[]:[] snapc=0=[]) v7 -- ?+4194934 0x7f4d78f01010
2012-12-21 12:40:00.772123 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] eval_repop repgather(0x7f4d7863bba0 3775'135 rep_tid=13299 wfack=25,31 wfdisk=25,31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) wants=ad
2012-12-21 12:40:00.772146 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] apply_repop applying update on repgather(0x7f4d7863bba0 3775'135 rep_tid=13299 wfack=25,31 wfdisk=25,31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4)
2012-12-21 12:40:00.775536 7f4d927fc700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] op_applied repgather(0x7f4d7863bba0 applying 3775'135 rep_tid=13299 wfack=25,31 wfdisk=25,31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4)
2012-12-21 12:40:00.775704 7f4d927fc700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] eval_repop repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack=31 wfdisk=25,31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) wants=ad
2012-12-21 12:40:00.807245 7f4d9cff9700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 luod=3775'134 lcod 3775'134 mlcod 3775'133 active+clean] op_commit repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack=31 wfdisk=25,31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4)
2012-12-21 12:40:00.807271 7f4d9cff9700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] eval_repop repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack=31 wfdisk=31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) wants=ad
2012-12-21 12:40:31.728982 7f4da5c69700 0 log [WRN] : slow request 30.965405 seconds old, received at 2012-12-21 12:40:00.763528: osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 currently waiting for sub ops
2012-12-21 12:40:32.522252 7f4d8b7fe700 1 -- 172.16.0.13:6816/15146 <== osd.31 172.16.0.12:6822/17200 6133 ==== osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ack, result = 0) v1 ==== 162+0+0 (2474129029 0 0) 0x7f4d500008c0 con 0x7f4d4c000b10
2012-12-21 12:40:32.522284 7f4d8b7fe700 20 osd.25 3775 _dispatch 0x7f4d500008c0 osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ack, result = 0) v1
2012-12-21 12:40:32.522390 7f4d8b7fe700 15 osd.25 3775 enqueue_op 0x7f4d442f69e0 osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ack, result = 0) v1
2012-12-21 12:40:32.522450 7f4d87ff7700 10 osd.25 3775 dequeue_op 0x7f4d442f69e0 osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ack, result = 0) v1 pg pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean]
2012-12-21 12:40:32.522533 7f4d87ff7700 7 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] repop_ack rep_tid 13299 op osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 result 0 ack_type 1 from osd.31
2012-12-21 12:40:32.522565 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] eval_repop repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack= wfdisk=31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) wants=ad
2012-12-21 12:40:32.522582 7f4d87ff7700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] sending ack on repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack= wfdisk=31 op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) 0x7f4d791ab010
2012-12-21 12:40:32.616813 7f4d8b7fe700 1 -- 172.16.0.13:6816/15146 <== osd.31 172.16.0.12:6822/17200 6138 ==== osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ondisk, result = 0) v1 ==== 162+0+0 (2032311727 0 0) 0x7f4d50001850 con 0x7f4d4c000b10
2012-12-21 12:40:32.616826 7f4d8b7fe700 20 osd.25 3775 _dispatch 0x7f4d50001850 osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ondisk, result = 0) v1
2012-12-21 12:40:32.616888 7f4d8b7fe700 15 osd.25 3775 enqueue_op 0x7f4d44d047a0 osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ondisk, result = 0) v1
2012-12-21 12:40:32.616918 7f4d877f6700 10 osd.25 3775 dequeue_op 0x7f4d44d047a0 osd_sub_op_reply(client.9501.0:744 6.95b 5d83495b/benchmark_data_KVM04_16365_object743/head//6 [] ondisk, result = 0) v1 pg pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean]
2012-12-21 12:40:32.616995 7f4d877f6700 7 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] repop_ack rep_tid 13299 op osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 result 0 ack_type 4 from osd.31
2012-12-21 12:40:32.617032 7f4d877f6700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] eval_repop repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack= wfdisk= op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) wants=ad
2012-12-21 12:40:32.617052 7f4d877f6700 15 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] log_op_stats osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4 inb 4194451 outb 0 rlat 31.759090 lat 31.853523
2012-12-21 12:40:32.617085 7f4d877f6700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'133 active+clean] sending commit on repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack= wfdisk= op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4) 0x7f4d816be810
2012-12-21 12:40:32.617121 7f4d877f6700 10 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'134 active+clean] removing repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack= wfdisk= op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4)
2012-12-21 12:40:32.617146 7f4d877f6700 20 osd.25 pg_epoch: 3775 pg[6.95b( v 3775'135 (0'0,3775'135] local-les=3774 n=15 ec=656 les/c 3774/3775 3773/3773/3773) [25,31] r=0 lpr=3773 mlcod 3775'134 active+clean] q front is repgather(0x7f4d7863bba0 applied 3775'135 rep_tid=13299 wfack= wfdisk= op=osd_op(client.9501.0:744 benchmark_data_KVM04_16365_object743 [write 0~4194304] 6.5d83495b) v4)
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-21 4:51 ` Matthew Anderson
@ 2012-12-21 16:02 ` Sage Weil
2013-01-09 8:52 ` Matthew Anderson
0 siblings, 1 reply; 13+ messages in thread
From: Sage Weil @ 2012-12-21 16:02 UTC (permalink / raw)
To: Matthew Anderson; +Cc: 'Mark Nelson', ceph-devel@vger.kernel.org
[-- Attachment #1: Type: TEXT/PLAIN, Size: 47800 bytes --]
On Fri, 21 Dec 2012, Matthew Anderson wrote:
> Hi Sage,
>
> I've tried to reproduce the error again with logging on every OSD and got the above. RADOS bench had stalled on a write request like the last time and the attached log is the grep'd OSD log (# cat osd.25.log | grep client.9501.0:744 > freeze.log) . The OSD that stalled was 25, pg map is below -
>
> # ceph pg map 6.5d83495b
> osdmap e3775 pg 6.5d83495b (6.95b) -> up [25,31] acting [25,31]
>
> I hope that's what you were after, if not just let me know
We're getting closer. The osd tried to send the reply. Can you reproduce
with 'debug ms = 20' on the osds too, and on the client side do
soemthing like
rados --debug-ms 20 --debug-objecter 20 --log-file /tmp/foo ...
Thanks!
sage
>
> Thanks again
> -Matt
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@inktank.com]
> Sent: Friday, 21 December 2012 1:14 AM
> To: Matthew Anderson
> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> Subject: RE: OSD's slow down to a crawl
>
> On Thu, 20 Dec 2012, Matthew Anderson wrote:
> > Hi Sage,
> >
> > Logs are attached. I took the osd logs from osd.24 as this is the
> > first osd in my SSD pool I've been testing with previously.
> >
> > The 4MB bench I was able to reproduce the fault by restarting my rbd
> > export which stalled after a few percent complete. When I ran the 4MB
> > bench it stalled early on and never received a response back from the
> > OSD and I terminated it after 60 seconds or so. I wasn't able to
> > reproduce the fault using the 4kb io size. The 4kb log should show
> > rados bench completing normally at a respectable speed of about 1MB/s.
>
> Let's drill into the hang.. up until that point things look okay.
>
> 2012-12-21 00:51:26.033622 7f6f3c042760 1 -- 172.16.0.13:0/1023886 --> 172.16.0.13:6813/22233 -- osd_op(client.9503.0:185 benchmark_data_KVM04_23886_object184 [write 0~4194304] 6.3ca4346e) v4 -- ?+0 0x171ea50 con 0x171a7e0
>
> Do you have a log for that OSD so we can see what happened there? It may also be that the replicated write is hung. If you do
>
> ceph pg map 6.3ca4346e
>
> you can see all OSDs storing that PG. And/or you can grep for
> client.9503.0:185 in 172.16.0.13:6813/22233's log and see whether the sub_op was sent.
>
> Thanks!
> sage
>
>
> >
> > Thanks
> > -Matt
> >
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Friday, 21 December 2012 12:30 AM
> > To: Matthew Anderson
> > Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> > Subject: RE: OSD's slow down to a crawl
> >
> > Can you do a similar test, but with full logging on?
> >
> > ceph tell osd.0 injectargs '--debug-ms 1 --debug-filestore 20
> > --debug-osd
> > 20 --debug-journal 20'
> > rados -p ssd bench 30 write -b 4096 -t 1 --log-file /tmp/foo
> > --debug-ms 1
> >
> > That will be a single IO in flight at a time and very easy to trace through the logs. If you can post the resulting log file (/tmp/foo and from osd.0), that would be awesome.
> >
> > Thanks!
> > sage
> >
> >
> >
> > On Thu, 20 Dec 2012, Matthew Anderson wrote:
> >
> > > # rados bench 60 write -t 256 -p ssd Maintaining 256 concurrent
> > > writes of 4194304 bytes for at least 60 seconds.
> > > Object prefix: benchmark_data_KVM03_12985
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 0 0 0 0 0 0 - 0
> > > 1 45 45 0 0 0 - 0
> > > 2 71 71 0 0 0 - 0
> > > 3 90 90 0 0 0 - 0
> > > 4 125 125 0 0 0 - 0
> > > 5 136 136 0 0 0 - 0
> > > 6 144 144 0 0 0 - 0
> > > 7 175 175 0 0 0 - 0
> > > 8 201 201 0 0 0 - 0
> > > 9 217 217 0 0 0 - 0
> > > 10 228 228 0 0 0 - 0
> > > 11 251 251 0 0 0 - 0
> > > 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> > > 13 255 271 16 4.92194 20 12.9795 11.9979
> > > 14 255 291 36 10.2833 80 13.1795 12.6478
> > > 15 255 301 46 12.2639 40 13.9424 12.9123
> > > 16 255 303 48 11.9973 8 14.0277 12.9579
> > > 17 255 321 66 15.526 72 15.1408 13.4616
> > > 18 255 345 90 19.9956 96 15.5673 13.9937
> > > 19 255 349 94 19.7851 16 15.707 14.0635
> > > 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 20 255 361 106 21.1953 48 16.4053 14.3121
> > > 21 255 367 112 21.3285 24 17.4263 14.4795
> > > 22 255 370 115 20.9043 12 17.3579 14.5544
> > > 23 255 387 132 22.9513 68 18.3185 15.0867
> > > 24 255 394 139 23.1614 28 18.2868 15.2504
> > > 25 255 404 149 23.8346 40 18.741 15.5048
> > > 26 255 410 155 23.8407 24 19.032 15.6358
> > > 27 255 410 155 22.9577 0 - 15.6358
> > > 28 255 410 155 22.1378 0 - 15.6358
> > > 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> > > 30 255 431 176 23.4614 80 22.6577 16.4822
> > > 31 255 447 192 24.7687 64 23.2345 17.0563
> > > 32 255 464 209 26.1192 68 23.3788 17.5859
> > > 33 255 485 230 27.8726 84 22.8924 18.1122
> > > 34 255 497 242 28.4642 48 23.563 18.3693
> > > 35 255 508 253 28.9078 44 23.0516 18.5935
> > > 36 255 514 259 28.7713 24 24.4415 18.7278
> > > 37 255 530 275 29.723 64 23.847 19.0424
> > > 38 255 541 286 30.0985 44 24.5757 19.2427
> > > 39 255 544 289 29.6344 12 24.6802 19.2984
> > > 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 40 255 544 289 28.8935 0 - 19.2984
> > > 41 255 544 289 28.1888 0 - 19.2984
> > > 42 255 544 289 27.5176 0 - 19.2984
> > > 43 255 544 289 26.8776 0 - 19.2984
> > > 44 255 544 289 26.2668 0 - 19.2984
> > > 45 255 544 289 25.6831 0 - 19.2984
> > > 46 255 544 289 25.1248 0 - 19.2984
> > > 47 255 544 289 24.5902 0 - 19.2984
> > > 48 255 544 289 24.0779 0 - 19.2984
> > > 49 255 544 289 23.5865 0 - 19.2984
> > > 50 255 544 289 23.1148 0 - 19.2984
> > > 51 255 544 289 22.6616 0 - 19.2984
> > > 52 255 544 289 22.2258 0 - 19.2984
> > > 53 255 544 289 21.8065 0 - 19.2984
> > > 54 255 544 289 21.4026 0 - 19.2984
> > > 55 255 544 289 21.0135 0 - 19.2984
> > > 56 255 544 289 20.6383 0 - 19.2984
> > > 57 255 544 289 20.2762 0 - 19.2984
> > > 58 255 544 289 19.9266 0 - 19.2984
> > > 59 255 544 289 19.5889 0 - 19.2984
> > > 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 60 255 544 289 19.2624 0 - 19.2984
> > > 61 255 544 289 18.9466 0 - 19.2984
> > > 62 255 544 289 18.641 0 - 19.2984
> > > 63 255 544 289 18.3452 0 - 19.2984
> > > 64 255 544 289 18.0585 0 - 19.2984
> > > 65 255 544 289 17.7807 0 - 19.2984
> > > 66 255 544 289 17.5113 0 - 19.2984
> > > 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> > > Total time run: 68.000740
> > > Total writes made: 545
> > > Write size: 4194304
> > > Bandwidth (MB/sec): 32.058
> > >
> > > Stddev Bandwidth: 26.8837
> > > Max bandwidth (MB/sec): 96
> > > Min bandwidth (MB/sec): 0
> > > Average Latency: 29.2801
> > > Stddev Latency: 11.9768
> > > Max latency: 52.6277
> > > Min latency: 11.4904
> > >
> > >
> > >
> > >
> > >
> > > ----Original Message-----
> > > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > > Sent: Thursday, 20 December 2012 11:59 PM
> > > To: Matthew Anderson
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: OSD's slow down to a crawl
> > >
> > > Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
> > >
> > > Mark
> > >
> > > On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> > > > Hi Mark,
> > > >
> > > > Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
> > > >
> > > > It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
> > > >
> > > > The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> > > > # ceph --admin-daemon ./ceph-osd.24.asok perf dump
> > > >
> > > > {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,"j
> > > > ou
> > > > rn
> > > > al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue_b
> > > > yt
> > > > es
> > > > ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"
> > > > su
> > > > m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":
> > > > 13
> > > > 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":
> > > > 20
> > > > 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":1710
> > > > 23
> > > > 25
> > > > 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
> > > > :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":
> > > > 17
> > > > 08
> > > > .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.2418500
> > > > 00
> > > > },
> > > > "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"
> > > > av
> > > > gc
> > > > ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
> > > > :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock"
> > > > :{
> > > > "w
> > > > ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::writeq_
> > > > lo
> > > > ck
> > > > ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
> > > > apply_lo
> > > ck":{"wai
> > > t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com_l
> > > oc
> > > k":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitManag
> > > er
> > > ::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"op
> > > _w
> > > ip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"
> > > op
> > > _latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r_o
> > > ut
> > > _bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.8336890
> > > 00
> > > },"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"
> > > sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.34968400
> > > 0}
> > > ,"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"avg
> > > co
> > > unt":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.0000
> > > 00
> > > 000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avgco
> > > un
> > > t":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"
> > > subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"
> > > subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":0,
> > > "s
> > > ubop_push_
> > > in_bytes"
> > > :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,"p
> > > us
> > > h":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops"
> > > :0
> > > ,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,"n
> > > um
> > > pg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat_f
> > > ro
> > > m_peers":0,"map_messages":44,"map_message_epochs":184,"map_message_e
> > > po
> > > ch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,"g
> > > et
> > > ":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":
> > > 20
> > > 868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"
> > > av
> > > gcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":
> > > 500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"
> > > take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"av
> > > gc
> > > ount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":
> > > {"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_fai
> > > l_
> > > fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,"p
> > > ut
> > > _sum":7955
> > > 46573,"wa
> > > it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_thrott
> > > le
> > > r-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":828011293,
> > > "g
> > > et_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put"
> > > :18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}},
> > > "t
> > > hrottle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"
> > > ge
> > > t":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":
> > > 0,
> > > "take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount"
> > > :0
> > > ,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{"v
> > > al
> > > ":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"
> > > get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":19
> > > 90
> > > 92,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_byt
> > > es
> > > ":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or_f
> > > ai
> > > l_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":37671,
> > > "p ut_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
> > > >
> > > > # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
> > > >
> > > > { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> > > > "received_at": "2012-12-20 23:34:42.012131",
> > > > "age": "75.428051",
> > > > "duration": "0.009988",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101498},
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:42.012317",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:42.012425",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:42.012678",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:42.012720",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:42.012815",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:34:42.012958",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:34:42.013033",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:34:42.013745",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:34:42.013803",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:34:42.017414",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:34:42.022066",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:34:42.022119",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:34:42.017048",
> > > > "age": "75.423134",
> > > > "duration": "0.000411",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:42.017218",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:42.017363",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:42.017401",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:42.017459",
> > > > "event": "done"}]},
> > > > { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> > > > "received_at": "2012-12-20 23:34:49.851021",
> > > > "age": "67.589161",
> > > > "duration": "0.007114",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101502},
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:49.851276",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:49.851429",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:49.851616",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:49.851648",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:49.851712",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:34:49.851770",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:34:49.851915",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:34:49.853956",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:34:49.853992",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:34:49.857016",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:34:49.858077",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:34:49.858135",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:34:49.856762",
> > > > "age": "67.583420",
> > > > "duration": "0.000308",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:49.856869",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:49.856969",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:49.857001",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:49.857070",
> > > > "event": "done"}]},
> > > > { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> > > > "received_at": "2012-12-20 23:35:34.351518",
> > > > "age": "23.088664",
> > > > "duration": "0.009430",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101504},
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:34.351714",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:34.351811",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:34.352030",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:34.352071",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:34.352146",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:35:34.352213",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:35:34.352287",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:35:34.355390",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:35:34.355475",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:35:34.356661",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:35:34.360894",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:35:34.360948",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:35:34.356286",
> > > > "age": "23.083896",
> > > > "duration": "0.000421",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:34.356489",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:34.356598",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:34.356640",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:34.356707",
> > > > "event": "done"}]},
> > > > { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> > > > "received_at": "2012-12-20 23:35:42.133103",
> > > > "age": "15.307079",
> > > > "duration": "0.008357",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101512},
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:42.133315",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:42.133427",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:42.133684",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:42.133723",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:42.133796",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:35:42.133908",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:35:42.133995",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:35:42.136965",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:35:42.137025",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:35:42.138456",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:35:42.141408",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:35:42.141460",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:35:42.138087",
> > > > "age": "15.302095",
> > > > "duration": "0.000410",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:42.138248",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:42.138363",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:42.138443",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:42.138497",
> > > > "event": "done"}]}]}
> > > >
> > > > # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
> > > > "osd.24",
> > > > "cluster": "ceph",
> > > > "none": "0\/5",
> > > > "lockdep": "0\/1",
> > > > "context": "0\/1",
> > > > "crush": "1\/1",
> > > > "mds": "1\/5",
> > > > "mds_balancer": "1\/5",
> > > > "mds_locker": "1\/5",
> > > > "mds_log": "1\/5",
> > > > "mds_log_expire": "1\/5",
> > > > "mds_migrator": "1\/5",
> > > > "buffer": "0\/1",
> > > > "timer": "0\/1",
> > > > "filer": "0\/1",
> > > > "striper": "0\/1",
> > > > "objecter": "0\/1",
> > > > "rados": "0\/5",
> > > > "rbd": "0\/5",
> > > > "journaler": "0\/5",
> > > > "objectcacher": "0\/5",
> > > > "client": "0\/5",
> > > > "osd": "0\/5",
> > > > "optracker": "0\/5",
> > > > "objclass": "0\/5",
> > > > "filestore": "1\/3",
> > > > "journal": "1\/3",
> > > > "ms": "0\/5",
> > > > "mon": "1\/5",
> > > > "monc": "0\/10",
> > > > "paxos": "0\/5",
> > > > "tp": "0\/5",
> > > > "auth": "1\/5",
> > > > "crypto": "1\/5",
> > > > "finisher": "1\/1",
> > > > "heartbeatmap": "1\/5",
> > > > "perfcounter": "1\/5",
> > > > "rgw": "1\/5",
> > > > "hadoop": "1\/5",
> > > > "javaclient": "1\/5",
> > > > "asok": "1\/5",
> > > > "throttle": "1\/1",
> > > > "host": "KVM04",
> > > > "fsid": "00000000-0000-0000-0000-000000000000",
> > > > "public_addr": ":\/0",
> > > > "cluster_addr": ":\/0",
> > > > "public_network": "",
> > > > "cluster_network": "",
> > > > "num_client": "1",
> > > > "monmap": "",
> > > > "mon_host": "",
> > > > "lockdep": "false",
> > > > "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> > > > "daemonize": "true",
> > > > "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> > > > "chdir": "\/",
> > > > "max_open_files": "0",
> > > > "fatal_signal_handlers": "true",
> > > > "log_file": "\/var\/log\/ceph\/osd.24.log",
> > > > "log_max_new": "1000",
> > > > "log_max_recent": "100000",
> > > > "log_to_stderr": "false",
> > > > "err_to_stderr": "true",
> > > > "log_to_syslog": "false",
> > > > "err_to_syslog": "false",
> > > > "log_flush_on_exit": "true",
> > > > "clog_to_monitors": "true",
> > > > "clog_to_syslog": "false",
> > > > "mon_cluster_log_to_syslog": "false",
> > > > "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> > > > "key": "",
> > > > "keyfile": "",
> > > > "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> > > > "heartbeat_interval": "5",
> > > > "heartbeat_file": "",
> > > > "perf": "true",
> > > > "ms_tcp_nodelay": "true",
> > > > "ms_initial_backoff": "0.2",
> > > > "ms_max_backoff": "15",
> > > > "ms_nocrc": "false",
> > > > "ms_die_on_bad_msg": "false",
> > > > "ms_dispatch_throttle_bytes": "104857600",
> > > > "ms_bind_ipv6": "false",
> > > > "ms_bind_port_min": "6800",
> > > > "ms_bind_port_max": "7100",
> > > > "ms_rwthread_stack_bytes": "1048576",
> > > > "ms_tcp_read_timeout": "900",
> > > > "ms_inject_socket_failures": "0",
> > > > "ms_inject_delay_type": "",
> > > > "ms_inject_delay_max": "1",
> > > > "ms_inject_delay_probability": "0",
> > > > "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> > > > "mon_initial_members": "",
> > > > "mon_sync_fs_threshold": "5",
> > > > "mon_tick_interval": "5",
> > > > "mon_subscribe_interval": "300",
> > > > "mon_osd_laggy_halflife": "3600",
> > > > "mon_osd_laggy_weight": "0.3",
> > > > "mon_osd_adjust_heartbeat_grace": "true",
> > > > "mon_osd_adjust_down_out_interval": "true",
> > > > "mon_osd_auto_mark_in": "false",
> > > > "mon_osd_auto_mark_auto_out_in": "true",
> > > > "mon_osd_auto_mark_new_in": "true",
> > > > "mon_osd_down_out_interval": "300",
> > > > "mon_osd_min_up_ratio": "0.3",
> > > > "mon_osd_min_in_ratio": "0.3",
> > > > "mon_lease": "5",
> > > > "mon_lease_renew_interval": "3",
> > > > "mon_lease_ack_timeout": "10",
> > > > "mon_clock_drift_allowed": "0.05",
> > > > "mon_clock_drift_warn_backoff": "5",
> > > > "mon_accept_timeout": "10",
> > > > "mon_pg_create_interval": "30",
> > > > "mon_pg_stuck_threshold": "300",
> > > > "mon_osd_full_ratio": "0.95",
> > > > "mon_osd_nearfull_ratio": "0.85",
> > > > "mon_globalid_prealloc": "100",
> > > > "mon_osd_report_timeout": "900",
> > > > "mon_force_standby_active": "true",
> > > > "mon_min_osdmap_epochs": "500",
> > > > "mon_max_pgmap_epochs": "500",
> > > > "mon_max_log_epochs": "500",
> > > > "mon_max_osd": "10000",
> > > > "mon_probe_timeout": "2",
> > > > "mon_slurp_timeout": "10",
> > > > "mon_slurp_bytes": "262144",
> > > > "mon_client_bytes": "104857600",
> > > > "mon_daemon_bytes": "419430400",
> > > > "mon_max_log_entries_per_event": "4096",
> > > > "paxos_max_join_drift": "10",
> > > > "paxos_propose_interval": "1",
> > > > "paxos_min_wait": "0.05",
> > > > "clock_offset": "0",
> > > > "auth_cluster_required": "none",
> > > > "auth_service_required": "none",
> > > > "auth_client_required": "none",
> > > > "auth_supported": "none",
> > > > "cephx_require_signatures": "false",
> > > > "cephx_sign_messages": "true",
> > > > "auth_mon_ticket_ttl": "43200",
> > > > "auth_service_ticket_ttl": "3600",
> > > > "mon_client_hunt_interval": "3",
> > > > "mon_client_ping_interval": "10",
> > > > "mon_client_max_log_entries_per_message": "1000",
> > > > "mon_max_pool_pg_num": "65536",
> > > > "client_cache_size": "16384",
> > > > "client_cache_mid": "0.75",
> > > > "client_use_random_mds": "false",
> > > > "client_mount_timeout": "30",
> > > > "client_tick_interval": "1",
> > > > "client_trace": "",
> > > > "client_readahead_min": "131072",
> > > > "client_readahead_max_bytes": "0",
> > > > "client_readahead_max_periods": "4",
> > > > "client_snapdir": ".snap",
> > > > "client_mountpoint": "\/",
> > > > "client_notify_timeout": "10",
> > > > "client_caps_release_delay": "5",
> > > > "client_oc": "true",
> > > > "client_oc_size": "209715200",
> > > > "client_oc_max_dirty": "104857600",
> > > > "client_oc_target_dirty": "8388608",
> > > > "client_oc_max_dirty_age": "5",
> > > > "client_oc_max_objects": "1000",
> > > > "client_debug_force_sync_read": "false",
> > > > "client_debug_inject_tick_delay": "0",
> > > > "fuse_use_invalidate_cb": "false",
> > > > "fuse_big_writes": "true",
> > > > "fuse_debug": "false",
> > > > "objecter_tick_interval": "5",
> > > > "objecter_timeout": "10",
> > > > "objecter_inflight_op_bytes": "104857600",
> > > > "objecter_inflight_ops": "1024",
> > > > "journaler_allow_split_entries": "true",
> > > > "journaler_write_head_interval": "15",
> > > > "journaler_prefetch_periods": "10",
> > > > "journaler_prezero_periods": "5",
> > > > "journaler_batch_interval": "0.001",
> > > > "journaler_batch_max": "0",
> > > > "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> > > > "mds_max_file_size": "1099511627776",
> > > > "mds_cache_size": "100000",
> > > > "mds_cache_mid": "0.7",
> > > > "mds_mem_max": "1048576",
> > > > "mds_dir_commit_ratio": "0.5",
> > > > "mds_dir_max_commit_size": "90",
> > > > "mds_decay_halflife": "5",
> > > > "mds_beacon_interval": "4",
> > > > "mds_beacon_grace": "15",
> > > > "mds_blacklist_interval": "1440",
> > > > "mds_session_timeout": "60",
> > > > "mds_session_autoclose": "300",
> > > > "mds_reconnect_timeout": "45",
> > > > "mds_tick_interval": "5",
> > > > "mds_dirstat_min_interval": "1",
> > > > "mds_scatter_nudge_interval": "5",
> > > > "mds_client_prealloc_inos": "1000",
> > > > "mds_early_reply": "true",
> > > > "mds_use_tmap": "true",
> > > > "mds_default_dir_hash": "2",
> > > > "mds_log": "true",
> > > > "mds_log_skip_corrupt_events": "false",
> > > > "mds_log_max_events": "-1",
> > > > "mds_log_max_segments": "30",
> > > > "mds_log_max_expiring": "20",
> > > > "mds_bal_sample_interval": "3",
> > > > "mds_bal_replicate_threshold": "8000",
> > > > "mds_bal_unreplicate_threshold": "0",
> > > > "mds_bal_frag": "false",
> > > > "mds_bal_split_size": "10000",
> > > > "mds_bal_split_rd": "25000",
> > > > "mds_bal_split_wr": "10000",
> > > > "mds_bal_split_bits": "3",
> > > > "mds_bal_merge_size": "50",
> > > > "mds_bal_merge_rd": "1000",
> > > > "mds_bal_merge_wr": "1000",
> > > > "mds_bal_interval": "10",
> > > > "mds_bal_fragment_interval": "5",
> > > > "mds_bal_idle_threshold": "0",
> > > > "mds_bal_max": "-1",
> > > > "mds_bal_max_until": "-1",
> > > > "mds_bal_mode": "0",
> > > > "mds_bal_min_rebalance": "0.1",
> > > > "mds_bal_min_start": "0.2",
> > > > "mds_bal_need_min": "0.8",
> > > > "mds_bal_need_max": "1.2",
> > > > "mds_bal_midchunk": "0.3",
> > > > "mds_bal_minchunk": "0.001",
> > > > "mds_bal_target_removal_min": "5",
> > > > "mds_bal_target_removal_max": "10",
> > > > "mds_replay_interval": "1",
> > > > "mds_shutdown_check": "0",
> > > > "mds_thrash_exports": "0",
> > > > "mds_thrash_fragments": "0",
> > > > "mds_dump_cache_on_map": "false",
> > > > "mds_dump_cache_after_rejoin": "false",
> > > > "mds_verify_scatter": "false",
> > > > "mds_debug_scatterstat": "false",
> > > > "mds_debug_frag": "false",
> > > > "mds_debug_auth_pins": "false",
> > > > "mds_debug_subtrees": "false",
> > > > "mds_kill_mdstable_at": "0",
> > > > "mds_kill_export_at": "0",
> > > > "mds_kill_import_at": "0",
> > > > "mds_kill_link_at": "0",
> > > > "mds_kill_rename_at": "0",
> > > > "mds_wipe_sessions": "false",
> > > > "mds_wipe_ino_prealloc": "false",
> > > > "mds_skip_ino": "0",
> > > > "max_mds": "1",
> > > > "mds_standby_for_name": "",
> > > > "mds_standby_for_rank": "-1",
> > > > "mds_standby_replay": "false",
> > > > "osd_auto_upgrade_tmap": "true",
> > > > "osd_tmapput_sets_uses_tmap": "false",
> > > > "osd_max_backfills": "10",
> > > > "osd_backfill_full_ratio": "0.85",
> > > > "osd_backfill_retry_interval": "10",
> > > > "osd_uuid": "00000000-0000-0000-0000-000000000000",
> > > > "osd_data": "\/srv\/ceph\/osd\/osd.24",
> > > > "osd_journal": "\/dev\/sda3",
> > > > "osd_journal_size": "5120",
> > > > "osd_max_write_size": "90",
> > > > "osd_max_pgls": "1024",
> > > > "osd_client_message_size_cap": "524288000",
> > > > "osd_pg_bits": "6",
> > > > "osd_pgp_bits": "6",
> > > > "osd_min_rep": "1",
> > > > "osd_max_rep": "10",
> > > > "osd_pool_default_crush_rule": "0",
> > > > "osd_pool_default_size": "2",
> > > > "osd_pool_default_min_size": "0",
> > > > "osd_pool_default_pg_num": "8",
> > > > "osd_pool_default_pgp_num": "8",
> > > > "osd_map_dedup": "true",
> > > > "osd_map_cache_size": "500",
> > > > "osd_map_message_max": "100",
> > > > "osd_op_threads": "2",
> > > > "osd_disk_threads": "1",
> > > > "osd_recovery_threads": "1",
> > > > "osd_recover_clone_overlap": "true",
> > > > "osd_backfill_scan_min": "64",
> > > > "osd_backfill_scan_max": "512",
> > > > "osd_op_thread_timeout": "30",
> > > > "osd_recovery_thread_timeout": "30",
> > > > "osd_snap_trim_thread_timeout": "3600",
> > > > "osd_scrub_thread_timeout": "60",
> > > > "osd_scrub_finalize_thread_timeout": "600",
> > > > "osd_remove_thread_timeout": "3600",
> > > > "osd_command_thread_timeout": "600",
> > > > "osd_age": "0.8",
> > > > "osd_age_time": "0",
> > > > "osd_heartbeat_addr": ":\/0",
> > > > "osd_heartbeat_interval": "6",
> > > > "osd_heartbeat_grace": "20",
> > > > "osd_mon_heartbeat_interval": "30",
> > > > "osd_mon_report_interval_max": "120",
> > > > "osd_mon_report_interval_min": "5",
> > > > "osd_mon_ack_timeout": "30",
> > > > "osd_min_down_reporters": "1",
> > > > "osd_min_down_reports": "3",
> > > > "osd_default_data_pool_replay_window": "45",
> > > > "osd_preserve_trimmed_log": "false",
> > > > "osd_auto_mark_unfound_lost": "false",
> > > > "osd_recovery_delay_start": "0",
> > > > "osd_recovery_max_active": "50",
> > > > "osd_recovery_max_chunk": "8388608",
> > > > "osd_recovery_forget_lost_objects": "false",
> > > > "osd_max_scrubs": "1",
> > > > "osd_scrub_load_threshold": "0.5",
> > > > "osd_scrub_min_interval": "300",
> > > > "osd_scrub_max_interval": "86400",
> > > > "osd_deep_scrub_interval": "604800",
> > > > "osd_deep_scrub_stride": "524288",
> > > > "osd_auto_weight": "false",
> > > > "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> > > > "osd_check_for_log_corruption": "false",
> > > > "osd_use_stale_snap": "false",
> > > > "osd_rollback_to_cluster_snap": "",
> > > > "osd_default_notify_timeout": "30",
> > > > "osd_kill_backfill_at": "0",
> > > > "osd_min_pg_log_entries": "1000",
> > > > "osd_op_complaint_time": "30",
> > > > "osd_command_max_records": "256",
> > > > "osd_op_log_threshold": "5",
> > > > "osd_verify_sparse_read_holes": "false",
> > > > "osd_debug_drop_ping_probability": "0",
> > > > "osd_debug_drop_ping_duration": "0",
> > > > "osd_debug_drop_pg_create_probability": "0",
> > > > "osd_debug_drop_pg_create_duration": "1",
> > > > "osd_debug_drop_op_probability": "0",
> > > > "osd_op_history_size": "20",
> > > > "osd_op_history_duration": "600",
> > > > "osd_target_transaction_size": "300",
> > > > "osd_client_op_priority": "63",
> > > > "osd_recovery_op_priority": "10",
> > > > "filestore": "false",
> > > > "filestore_index_retry_probability": "0",
> > > > "filestore_debug_omap_check": "false",
> > > > "filestore_xattr_use_omap": "false",
> > > > "filestore_max_inline_xattr_size": "512",
> > > > "filestore_max_inline_xattrs": "2",
> > > > "filestore_max_sync_interval": "5",
> > > > "filestore_min_sync_interval": "0.01",
> > > > "filestore_btrfs_snap": "true",
> > > > "filestore_btrfs_clone_range": "true",
> > > > "filestore_fsync_flushes_journal_data": "false",
> > > > "filestore_fiemap": "false",
> > > > "filestore_flusher": "true",
> > > > "filestore_flusher_max_fds": "512",
> > > > "filestore_flush_min": "65536",
> > > > "filestore_sync_flush": "false",
> > > > "filestore_journal_parallel": "false",
> > > > "filestore_journal_writeahead": "false",
> > > > "filestore_journal_trailing": "false",
> > > > "filestore_queue_max_ops": "500",
> > > > "filestore_queue_max_bytes": "104857600",
> > > > "filestore_queue_committing_max_ops": "500",
> > > > "filestore_queue_committing_max_bytes": "104857600",
> > > > "filestore_op_threads": "2",
> > > > "filestore_op_thread_timeout": "60",
> > > > "filestore_op_thread_suicide_timeout": "180",
> > > > "filestore_commit_timeout": "600",
> > > > "filestore_fiemap_threshold": "4096",
> > > > "filestore_merge_threshold": "10",
> > > > "filestore_split_multiple": "2",
> > > > "filestore_update_to": "1000",
> > > > "filestore_blackhole": "false",
> > > > "filestore_dump_file": "",
> > > > "filestore_kill_at": "0",
> > > > "filestore_fail_eio": "true",
> > > > "journal_dio": "true",
> > > > "journal_aio": "false",
> > > > "journal_block_align": "true",
> > > > "journal_max_write_bytes": "10485760",
> > > > "journal_max_write_entries": "100",
> > > > "journal_queue_max_ops": "500",
> > > > "journal_queue_max_bytes": "104857600",
> > > > "journal_align_min_size": "65536",
> > > > "journal_replay_from": "0",
> > > > "journal_zero_on_create": "false",
> > > > "rbd_cache": "false",
> > > > "rbd_cache_size": "33554432",
> > > > "rbd_cache_max_dirty": "25165824",
> > > > "rbd_cache_target_dirty": "16777216",
> > > > "rbd_cache_max_dirty_age": "1",
> > > > "nss_db_path": "",
> > > > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> > > > "rgw_enable_apis": "s3, swift, swift_auth, admin",
> > > > "rgw_cache_enabled": "true",
> > > > "rgw_cache_lru_size": "10000",
> > > > "rgw_socket_path": "",
> > > > "rgw_dns_name": "",
> > > > "rgw_swift_url": "",
> > > > "rgw_swift_url_prefix": "swift",
> > > > "rgw_swift_auth_url": "",
> > > > "rgw_swift_auth_entry": "auth",
> > > > "rgw_keystone_url": "",
> > > > "rgw_keystone_admin_token": "",
> > > > "rgw_keystone_accepted_roles": "Member, admin",
> > > > "rgw_keystone_token_cache_size": "10000",
> > > > "rgw_keystone_revocation_interval": "900",
> > > > "rgw_admin_entry": "admin",
> > > > "rgw_enforce_swift_acls": "true",
> > > > "rgw_swift_token_expiration": "86400",
> > > > "rgw_print_continue": "true",
> > > > "rgw_remote_addr_param": "REMOTE_ADDR",
> > > > "rgw_op_thread_timeout": "600",
> > > > "rgw_op_thread_suicide_timeout": "0",
> > > > "rgw_thread_pool_size": "100",
> > > > "rgw_num_control_oids": "8",
> > > > "rgw_cluster_root_pool": ".rgw.root",
> > > > "rgw_log_nonexistent_bucket": "false",
> > > > "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > > > "rgw_log_object_name_utc": "false",
> > > > "rgw_usage_max_shards": "32",
> > > > "rgw_usage_max_user_shards": "1",
> > > > "rgw_enable_ops_log": "true",
> > > > "rgw_enable_usage_log": "true",
> > > > "rgw_ops_log_rados": "true",
> > > > "rgw_ops_log_socket_path": "",
> > > > "rgw_ops_log_data_backlog": "5242880",
> > > > "rgw_usage_log_flush_threshold": "1024",
> > > > "rgw_usage_log_tick_interval": "30",
> > > > "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> > > > "rgw_intent_log_object_name_utc": "false",
> > > > "rgw_init_timeout": "30",
> > > > "rgw_mime_types_file": "\/etc\/mime.types",
> > > > "rgw_gc_max_objs": "32",
> > > > "rgw_gc_obj_min_wait": "7200",
> > > > "rgw_gc_processor_max_time": "3600",
> > > > "rgw_gc_processor_period": "3600",
> > > > "rgw_s3_success_create_obj_status": "0",
> > > > "rgw_resolve_cname": "false",
> > > > "rgw_obj_stripe_size": "4194304",
> > > > "rgw_extended_http_attrs": "",
> > > > "mutex_perf_counter": "false",
> > > > "internal_safe_to_start_threads": "true"}
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > > > Sent: Thursday, 20 December 2012 11:28 PM
> > > > To: Matthew Anderson
> > > > Cc: ceph-devel@vger.kernel.org
> > > > Subject: Re: OSD's slow down to a crawl
> > > >
> > > > Hi Matt,
> > > >
> > > > Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> > > > Sebastien Han made a nice little blog entry describing it:
> > > >
> > > > http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
> > > >
> > > > You can get a lot of information about the current state of things.
> > > >
> > > > Mark
> > > >
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > in the body of a message to majordomo@vger.kernel.org More majordomo
> > > info at http://vger.kernel.org/majordomo-info.html
> > >
> > >
> >
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: OSD's slow down to a crawl
2012-12-21 16:02 ` Sage Weil
@ 2013-01-09 8:52 ` Matthew Anderson
2013-01-09 14:09 ` Mark Nelson
0 siblings, 1 reply; 13+ messages in thread
From: Matthew Anderson @ 2013-01-09 8:52 UTC (permalink / raw)
To: 'Sage Weil'; +Cc: 'Mark Nelson', ceph-devel@vger.kernel.org
Hi Sage,
Sorry for the late follow up, I've been on a bit of a testing rampage and managed to somewhat sort the problem.
Most of the problems appears to be from the 3.7.1 kernel. It seems to have a fairly big issue with its networking stack that was causing Ceph's network operations to hang. Moving back to a 3.6.8 kernel fixed this up. I don't know the full extent of the problem but it was reported on Phoronix briefly here - http://www.phoronix.com/scan.php?page=news_item&px=MTI2Nzc
The second issue was BTRFS on both the 3.7 and 3.6.8 kernels. After running a long rados bench (10 minutes) on a fresh cluster it would often slow down significantly by going from 250MB/s down to a 50MB/s average. Latency also increased dramatically. Restarting the OSD's fixes the issue but after a while it slows right down again. In the end I re-formatted the cluster using XFS (and also EXT4 for benchmarks) and there wasn't a single issue. I had rados bench running for over 30 minutes from another machine and there wasn't a single issue.
At thisstage I need to start moving into production with XFS. My test cluster arrives in a few weeks so I should be able to come back to the BTRFS issue later on as it would be very handy to have compression working.
Thanks again for your help
-Matt
-----Original Message-----
From: Sage Weil [mailto:sage@inktank.com]
Sent: Saturday, 22 December 2012 12:02 AM
To: Matthew Anderson
Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
Subject: RE: OSD's slow down to a crawl
On Fri, 21 Dec 2012, Matthew Anderson wrote:
> Hi Sage,
>
> I've tried to reproduce the error again with logging on every OSD and
> got the above. RADOS bench had stalled on a write request like the
> last time and the attached log is the grep'd OSD log (# cat osd.25.log
> | grep client.9501.0:744 > freeze.log) . The OSD that stalled was 25,
> pg map is below -
>
> # ceph pg map 6.5d83495b
> osdmap e3775 pg 6.5d83495b (6.95b) -> up [25,31] acting [25,31]
>
> I hope that's what you were after, if not just let me know
We're getting closer. The osd tried to send the reply. Can you reproduce with 'debug ms = 20' on the osds too, and on the client side do soemthing like
rados --debug-ms 20 --debug-objecter 20 --log-file /tmp/foo ...
Thanks!
sage
>
> Thanks again
> -Matt
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@inktank.com]
> Sent: Friday, 21 December 2012 1:14 AM
> To: Matthew Anderson
> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> Subject: RE: OSD's slow down to a crawl
>
> On Thu, 20 Dec 2012, Matthew Anderson wrote:
> > Hi Sage,
> >
> > Logs are attached. I took the osd logs from osd.24 as this is the
> > first osd in my SSD pool I've been testing with previously.
> >
> > The 4MB bench I was able to reproduce the fault by restarting my rbd
> > export which stalled after a few percent complete. When I ran the
> > 4MB bench it stalled early on and never received a response back
> > from the OSD and I terminated it after 60 seconds or so. I wasn't
> > able to reproduce the fault using the 4kb io size. The 4kb log
> > should show rados bench completing normally at a respectable speed of about 1MB/s.
>
> Let's drill into the hang.. up until that point things look okay.
>
> 2012-12-21 00:51:26.033622 7f6f3c042760 1 -- 172.16.0.13:0/1023886
> --> 172.16.0.13:6813/22233 -- osd_op(client.9503.0:185
> benchmark_data_KVM04_23886_object184 [write 0~4194304] 6.3ca4346e) v4
> -- ?+0 0x171ea50 con 0x171a7e0
>
> Do you have a log for that OSD so we can see what happened there? It
> may also be that the replicated write is hung. If you do
>
> ceph pg map 6.3ca4346e
>
> you can see all OSDs storing that PG. And/or you can grep for
> client.9503.0:185 in 172.16.0.13:6813/22233's log and see whether the sub_op was sent.
>
> Thanks!
> sage
>
>
> >
> > Thanks
> > -Matt
> >
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Friday, 21 December 2012 12:30 AM
> > To: Matthew Anderson
> > Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> > Subject: RE: OSD's slow down to a crawl
> >
> > Can you do a similar test, but with full logging on?
> >
> > ceph tell osd.0 injectargs '--debug-ms 1 --debug-filestore 20
> > --debug-osd
> > 20 --debug-journal 20'
> > rados -p ssd bench 30 write -b 4096 -t 1 --log-file /tmp/foo
> > --debug-ms 1
> >
> > That will be a single IO in flight at a time and very easy to trace through the logs. If you can post the resulting log file (/tmp/foo and from osd.0), that would be awesome.
> >
> > Thanks!
> > sage
> >
> >
> >
> > On Thu, 20 Dec 2012, Matthew Anderson wrote:
> >
> > > # rados bench 60 write -t 256 -p ssd Maintaining 256 concurrent
> > > writes of 4194304 bytes for at least 60 seconds.
> > > Object prefix: benchmark_data_KVM03_12985
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 0 0 0 0 0 0 - 0
> > > 1 45 45 0 0 0 - 0
> > > 2 71 71 0 0 0 - 0
> > > 3 90 90 0 0 0 - 0
> > > 4 125 125 0 0 0 - 0
> > > 5 136 136 0 0 0 - 0
> > > 6 144 144 0 0 0 - 0
> > > 7 175 175 0 0 0 - 0
> > > 8 201 201 0 0 0 - 0
> > > 9 217 217 0 0 0 - 0
> > > 10 228 228 0 0 0 - 0
> > > 11 251 251 0 0 0 - 0
> > > 12 255 266 11 3.66581 3.66667 11.7027 11.6013
> > > 13 255 271 16 4.92194 20 12.9795 11.9979
> > > 14 255 291 36 10.2833 80 13.1795 12.6478
> > > 15 255 301 46 12.2639 40 13.9424 12.9123
> > > 16 255 303 48 11.9973 8 14.0277 12.9579
> > > 17 255 321 66 15.526 72 15.1408 13.4616
> > > 18 255 345 90 19.9956 96 15.5673 13.9937
> > > 19 255 349 94 19.7851 16 15.707 14.0635
> > > 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 20 255 361 106 21.1953 48 16.4053 14.3121
> > > 21 255 367 112 21.3285 24 17.4263 14.4795
> > > 22 255 370 115 20.9043 12 17.3579 14.5544
> > > 23 255 387 132 22.9513 68 18.3185 15.0867
> > > 24 255 394 139 23.1614 28 18.2868 15.2504
> > > 25 255 404 149 23.8346 40 18.741 15.5048
> > > 26 255 410 155 23.8407 24 19.032 15.6358
> > > 27 255 410 155 22.9577 0 - 15.6358
> > > 28 255 410 155 22.1378 0 - 15.6358
> > > 29 255 411 156 21.5124 1.33333 22.6927 15.6811
> > > 30 255 431 176 23.4614 80 22.6577 16.4822
> > > 31 255 447 192 24.7687 64 23.2345 17.0563
> > > 32 255 464 209 26.1192 68 23.3788 17.5859
> > > 33 255 485 230 27.8726 84 22.8924 18.1122
> > > 34 255 497 242 28.4642 48 23.563 18.3693
> > > 35 255 508 253 28.9078 44 23.0516 18.5935
> > > 36 255 514 259 28.7713 24 24.4415 18.7278
> > > 37 255 530 275 29.723 64 23.847 19.0424
> > > 38 255 541 286 30.0985 44 24.5757 19.2427
> > > 39 255 544 289 29.6344 12 24.6802 19.2984
> > > 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 40 255 544 289 28.8935 0 - 19.2984
> > > 41 255 544 289 28.1888 0 - 19.2984
> > > 42 255 544 289 27.5176 0 - 19.2984
> > > 43 255 544 289 26.8776 0 - 19.2984
> > > 44 255 544 289 26.2668 0 - 19.2984
> > > 45 255 544 289 25.6831 0 - 19.2984
> > > 46 255 544 289 25.1248 0 - 19.2984
> > > 47 255 544 289 24.5902 0 - 19.2984
> > > 48 255 544 289 24.0779 0 - 19.2984
> > > 49 255 544 289 23.5865 0 - 19.2984
> > > 50 255 544 289 23.1148 0 - 19.2984
> > > 51 255 544 289 22.6616 0 - 19.2984
> > > 52 255 544 289 22.2258 0 - 19.2984
> > > 53 255 544 289 21.8065 0 - 19.2984
> > > 54 255 544 289 21.4026 0 - 19.2984
> > > 55 255 544 289 21.0135 0 - 19.2984
> > > 56 255 544 289 20.6383 0 - 19.2984
> > > 57 255 544 289 20.2762 0 - 19.2984
> > > 58 255 544 289 19.9266 0 - 19.2984
> > > 59 255 544 289 19.5889 0 - 19.2984
> > > 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
> > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> > > 60 255 544 289 19.2624 0 - 19.2984
> > > 61 255 544 289 18.9466 0 - 19.2984
> > > 62 255 544 289 18.641 0 - 19.2984
> > > 63 255 544 289 18.3452 0 - 19.2984
> > > 64 255 544 289 18.0585 0 - 19.2984
> > > 65 255 544 289 17.7807 0 - 19.2984
> > > 66 255 544 289 17.5113 0 - 19.2984
> > > 67 133 545 412 24.5916 17.5714 37.3574 27.4834
> > > Total time run: 68.000740
> > > Total writes made: 545
> > > Write size: 4194304
> > > Bandwidth (MB/sec): 32.058
> > >
> > > Stddev Bandwidth: 26.8837
> > > Max bandwidth (MB/sec): 96
> > > Min bandwidth (MB/sec): 0
> > > Average Latency: 29.2801
> > > Stddev Latency: 11.9768
> > > Max latency: 52.6277
> > > Min latency: 11.4904
> > >
> > >
> > >
> > >
> > >
> > > ----Original Message-----
> > > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > > Sent: Thursday, 20 December 2012 11:59 PM
> > > To: Matthew Anderson
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: OSD's slow down to a crawl
> > >
> > > Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
> > >
> > > Mark
> > >
> > > On 12/20/2012 09:51 AM, Matthew Anderson wrote:
> > > > Hi Mark,
> > > >
> > > > Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
> > > >
> > > > It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
> > > >
> > > > The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
> > > > # ceph --admin-daemon ./ceph-osd.24.asok perf dump
> > > >
> > > > {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,
> > > > "j
> > > > ou
> > > > rn
> > > > al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue
> > > > _b
> > > > yt
> > > > es
> > > > ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"
> > > > su
> > > > m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":
> > > > 13
> > > > 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":
> > > > 20
> > > > 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":17
> > > > 10
> > > > 23
> > > > 25
> > > > 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
> > > > :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":
> > > > 17
> > > > 08
> > > > .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.24185
> > > > 00
> > > > 00
> > > > },
> > > > "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"
> > > > av
> > > > gc
> > > > ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
> > > > :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock"
> > > > :{
> > > > "w
> > > > ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write
> > > > q_
> > > > lo
> > > > ck
> > > > ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
> > > > apply_lo
> > > ck":{"wai
> > > t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com
> > > _l
> > > oc
> > > k":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitMan
> > > ag
> > > er
> > > ::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"
> > > op
> > > _w
> > > ip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"
> > > op
> > > _latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r
> > > _o
> > > ut
> > > _bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.83368
> > > 90
> > > 00
> > > },"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"
> > > sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684
> > > 00
> > > 0}
> > > ,"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"a
> > > vg
> > > co
> > > unt":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.00
> > > 00
> > > 00
> > > 000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avg
> > > co
> > > un
> > > t":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"
> > > subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"
> > > subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":
> > > 0,
> > > "s
> > > ubop_push_
> > > in_bytes"
> > > :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,
> > > "p
> > > us
> > > h":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops"
> > > :0
> > > ,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,
> > > "n
> > > um
> > > pg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat
> > > _f
> > > ro
> > > m_peers":0,"map_messages":44,"map_message_epochs":184,"map_message
> > > _e
> > > po
> > > ch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,
> > > "g
> > > et
> > > ":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":
> > > 20
> > > 868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"
> > > av
> > > gcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":
> > > 500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"
> > > take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"
> > > av
> > > gc
> > > ount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":
> > > {"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_f
> > > ai
> > > l_
> > > fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,
> > > "p
> > > ut
> > > _sum":7955
> > > 46573,"wa
> > > it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_thro
> > > tt
> > > le
> > > r-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":82801129
> > > 3,
> > > "g
> > > et_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put"
> > > :18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}
> > > },
> > > "t
> > > hrottle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"
> > > ge
> > > t":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":
> > > 0,
> > > "take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount"
> > > :0
> > > ,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{
> > > "v
> > > al
> > > ":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"
> > > get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":
> > > 19
> > > 90
> > > 92,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_b
> > > yt
> > > es
> > > ":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or
> > > _f
> > > ai
> > > l_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":3767
> > > 1, "p ut_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
> > > >
> > > > # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
> > > >
> > > > { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
> > > > "received_at": "2012-12-20 23:34:42.012131",
> > > > "age": "75.428051",
> > > > "duration": "0.009988",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101498},
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:42.012317",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:42.012425",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:42.012678",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:42.012720",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:42.012815",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:34:42.012958",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:34:42.013033",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:34:42.013745",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:34:42.013803",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:34:42.017414",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:34:42.022066",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:34:42.022119",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:34:42.017048",
> > > > "age": "75.423134",
> > > > "duration": "0.000411",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:42.017218",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:42.017363",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:42.017401",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:42.017459",
> > > > "event": "done"}]},
> > > > { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
> > > > "received_at": "2012-12-20 23:34:49.851021",
> > > > "age": "67.589161",
> > > > "duration": "0.007114",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101502},
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:49.851276",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:49.851429",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:49.851616",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:49.851648",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:49.851712",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:34:49.851770",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:34:49.851915",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:34:49.853956",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:34:49.853992",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:34:49.857016",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:34:49.858077",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:34:49.858135",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:34:49.856762",
> > > > "age": "67.583420",
> > > > "duration": "0.000308",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:34:49.856869",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:34:49.856969",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:34:49.857001",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:34:49.857070",
> > > > "event": "done"}]},
> > > > { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
> > > > "received_at": "2012-12-20 23:35:34.351518",
> > > > "age": "23.088664",
> > > > "duration": "0.009430",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101504},
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:34.351714",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:34.351811",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:34.352030",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:34.352071",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:34.352146",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:35:34.352213",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:35:34.352287",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:35:34.355390",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:35:34.355475",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:35:34.356661",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:35:34.360894",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:35:34.360948",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:35:34.356286",
> > > > "age": "23.083896",
> > > > "duration": "0.000421",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:34.356489",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:34.356598",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:34.356640",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:34.356707",
> > > > "event": "done"}]},
> > > > { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
> > > > "received_at": "2012-12-20 23:35:42.133103",
> > > > "age": "15.307079",
> > > > "duration": "0.008357",
> > > > "flag_point": "waiting for sub ops",
> > > > "client_info": { "client": "client.7636",
> > > > "tid": 101512},
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:42.133315",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:42.133427",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:42.133684",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:42.133723",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:42.133796",
> > > > "event": "sub_op_sent"},
> > > > { "time": "2012-12-20 23:35:42.133908",
> > > > "event": "commit_queued_for_journal_write"},
> > > > { "time": "2012-12-20 23:35:42.133995",
> > > > "event": "write_thread_in_journal_buffer"},
> > > > { "time": "2012-12-20 23:35:42.136965",
> > > > "event": "journaled_completion_queued"},
> > > > { "time": "2012-12-20 23:35:42.137025",
> > > > "event": "op_commit"},
> > > > { "time": "2012-12-20 23:35:42.138456",
> > > > "event": "sub_op_commit_rec"},
> > > > { "time": "2012-12-20 23:35:42.141408",
> > > > "event": "op_applied"},
> > > > { "time": "2012-12-20 23:35:42.141460",
> > > > "event": "done"}]},
> > > > { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
> > > > "received_at": "2012-12-20 23:35:42.138087",
> > > > "age": "15.302095",
> > > > "duration": "0.000410",
> > > > "flag_point": "started",
> > > > "events": [
> > > > { "time": "2012-12-20 23:35:42.138248",
> > > > "event": "waiting_for_osdmap"},
> > > > { "time": "2012-12-20 23:35:42.138363",
> > > > "event": "reached_pg"},
> > > > { "time": "2012-12-20 23:35:42.138443",
> > > > "event": "started"},
> > > > { "time": "2012-12-20 23:35:42.138497",
> > > > "event": "done"}]}]}
> > > >
> > > > # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
> > > > "osd.24",
> > > > "cluster": "ceph",
> > > > "none": "0\/5",
> > > > "lockdep": "0\/1",
> > > > "context": "0\/1",
> > > > "crush": "1\/1",
> > > > "mds": "1\/5",
> > > > "mds_balancer": "1\/5",
> > > > "mds_locker": "1\/5",
> > > > "mds_log": "1\/5",
> > > > "mds_log_expire": "1\/5",
> > > > "mds_migrator": "1\/5",
> > > > "buffer": "0\/1",
> > > > "timer": "0\/1",
> > > > "filer": "0\/1",
> > > > "striper": "0\/1",
> > > > "objecter": "0\/1",
> > > > "rados": "0\/5",
> > > > "rbd": "0\/5",
> > > > "journaler": "0\/5",
> > > > "objectcacher": "0\/5",
> > > > "client": "0\/5",
> > > > "osd": "0\/5",
> > > > "optracker": "0\/5",
> > > > "objclass": "0\/5",
> > > > "filestore": "1\/3",
> > > > "journal": "1\/3",
> > > > "ms": "0\/5",
> > > > "mon": "1\/5",
> > > > "monc": "0\/10",
> > > > "paxos": "0\/5",
> > > > "tp": "0\/5",
> > > > "auth": "1\/5",
> > > > "crypto": "1\/5",
> > > > "finisher": "1\/1",
> > > > "heartbeatmap": "1\/5",
> > > > "perfcounter": "1\/5",
> > > > "rgw": "1\/5",
> > > > "hadoop": "1\/5",
> > > > "javaclient": "1\/5",
> > > > "asok": "1\/5",
> > > > "throttle": "1\/1",
> > > > "host": "KVM04",
> > > > "fsid": "00000000-0000-0000-0000-000000000000",
> > > > "public_addr": ":\/0",
> > > > "cluster_addr": ":\/0",
> > > > "public_network": "",
> > > > "cluster_network": "",
> > > > "num_client": "1",
> > > > "monmap": "",
> > > > "mon_host": "",
> > > > "lockdep": "false",
> > > > "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
> > > > "daemonize": "true",
> > > > "pid_file": "\/var\/run\/ceph\/osd.24.pid",
> > > > "chdir": "\/",
> > > > "max_open_files": "0",
> > > > "fatal_signal_handlers": "true",
> > > > "log_file": "\/var\/log\/ceph\/osd.24.log",
> > > > "log_max_new": "1000",
> > > > "log_max_recent": "100000",
> > > > "log_to_stderr": "false",
> > > > "err_to_stderr": "true",
> > > > "log_to_syslog": "false",
> > > > "err_to_syslog": "false",
> > > > "log_flush_on_exit": "true",
> > > > "clog_to_monitors": "true",
> > > > "clog_to_syslog": "false",
> > > > "mon_cluster_log_to_syslog": "false",
> > > > "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
> > > > "key": "",
> > > > "keyfile": "",
> > > > "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
> > > > "heartbeat_interval": "5",
> > > > "heartbeat_file": "",
> > > > "perf": "true",
> > > > "ms_tcp_nodelay": "true",
> > > > "ms_initial_backoff": "0.2",
> > > > "ms_max_backoff": "15",
> > > > "ms_nocrc": "false",
> > > > "ms_die_on_bad_msg": "false",
> > > > "ms_dispatch_throttle_bytes": "104857600",
> > > > "ms_bind_ipv6": "false",
> > > > "ms_bind_port_min": "6800",
> > > > "ms_bind_port_max": "7100",
> > > > "ms_rwthread_stack_bytes": "1048576",
> > > > "ms_tcp_read_timeout": "900",
> > > > "ms_inject_socket_failures": "0",
> > > > "ms_inject_delay_type": "",
> > > > "ms_inject_delay_max": "1",
> > > > "ms_inject_delay_probability": "0",
> > > > "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
> > > > "mon_initial_members": "",
> > > > "mon_sync_fs_threshold": "5",
> > > > "mon_tick_interval": "5",
> > > > "mon_subscribe_interval": "300",
> > > > "mon_osd_laggy_halflife": "3600",
> > > > "mon_osd_laggy_weight": "0.3",
> > > > "mon_osd_adjust_heartbeat_grace": "true",
> > > > "mon_osd_adjust_down_out_interval": "true",
> > > > "mon_osd_auto_mark_in": "false",
> > > > "mon_osd_auto_mark_auto_out_in": "true",
> > > > "mon_osd_auto_mark_new_in": "true",
> > > > "mon_osd_down_out_interval": "300",
> > > > "mon_osd_min_up_ratio": "0.3",
> > > > "mon_osd_min_in_ratio": "0.3",
> > > > "mon_lease": "5",
> > > > "mon_lease_renew_interval": "3",
> > > > "mon_lease_ack_timeout": "10",
> > > > "mon_clock_drift_allowed": "0.05",
> > > > "mon_clock_drift_warn_backoff": "5",
> > > > "mon_accept_timeout": "10",
> > > > "mon_pg_create_interval": "30",
> > > > "mon_pg_stuck_threshold": "300",
> > > > "mon_osd_full_ratio": "0.95",
> > > > "mon_osd_nearfull_ratio": "0.85",
> > > > "mon_globalid_prealloc": "100",
> > > > "mon_osd_report_timeout": "900",
> > > > "mon_force_standby_active": "true",
> > > > "mon_min_osdmap_epochs": "500",
> > > > "mon_max_pgmap_epochs": "500",
> > > > "mon_max_log_epochs": "500",
> > > > "mon_max_osd": "10000",
> > > > "mon_probe_timeout": "2",
> > > > "mon_slurp_timeout": "10",
> > > > "mon_slurp_bytes": "262144",
> > > > "mon_client_bytes": "104857600",
> > > > "mon_daemon_bytes": "419430400",
> > > > "mon_max_log_entries_per_event": "4096",
> > > > "paxos_max_join_drift": "10",
> > > > "paxos_propose_interval": "1",
> > > > "paxos_min_wait": "0.05",
> > > > "clock_offset": "0",
> > > > "auth_cluster_required": "none",
> > > > "auth_service_required": "none",
> > > > "auth_client_required": "none",
> > > > "auth_supported": "none",
> > > > "cephx_require_signatures": "false",
> > > > "cephx_sign_messages": "true",
> > > > "auth_mon_ticket_ttl": "43200",
> > > > "auth_service_ticket_ttl": "3600",
> > > > "mon_client_hunt_interval": "3",
> > > > "mon_client_ping_interval": "10",
> > > > "mon_client_max_log_entries_per_message": "1000",
> > > > "mon_max_pool_pg_num": "65536",
> > > > "client_cache_size": "16384",
> > > > "client_cache_mid": "0.75",
> > > > "client_use_random_mds": "false",
> > > > "client_mount_timeout": "30",
> > > > "client_tick_interval": "1",
> > > > "client_trace": "",
> > > > "client_readahead_min": "131072",
> > > > "client_readahead_max_bytes": "0",
> > > > "client_readahead_max_periods": "4",
> > > > "client_snapdir": ".snap",
> > > > "client_mountpoint": "\/",
> > > > "client_notify_timeout": "10",
> > > > "client_caps_release_delay": "5",
> > > > "client_oc": "true",
> > > > "client_oc_size": "209715200",
> > > > "client_oc_max_dirty": "104857600",
> > > > "client_oc_target_dirty": "8388608",
> > > > "client_oc_max_dirty_age": "5",
> > > > "client_oc_max_objects": "1000",
> > > > "client_debug_force_sync_read": "false",
> > > > "client_debug_inject_tick_delay": "0",
> > > > "fuse_use_invalidate_cb": "false",
> > > > "fuse_big_writes": "true",
> > > > "fuse_debug": "false",
> > > > "objecter_tick_interval": "5",
> > > > "objecter_timeout": "10",
> > > > "objecter_inflight_op_bytes": "104857600",
> > > > "objecter_inflight_ops": "1024",
> > > > "journaler_allow_split_entries": "true",
> > > > "journaler_write_head_interval": "15",
> > > > "journaler_prefetch_periods": "10",
> > > > "journaler_prezero_periods": "5",
> > > > "journaler_batch_interval": "0.001",
> > > > "journaler_batch_max": "0",
> > > > "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
> > > > "mds_max_file_size": "1099511627776",
> > > > "mds_cache_size": "100000",
> > > > "mds_cache_mid": "0.7",
> > > > "mds_mem_max": "1048576",
> > > > "mds_dir_commit_ratio": "0.5",
> > > > "mds_dir_max_commit_size": "90",
> > > > "mds_decay_halflife": "5",
> > > > "mds_beacon_interval": "4",
> > > > "mds_beacon_grace": "15",
> > > > "mds_blacklist_interval": "1440",
> > > > "mds_session_timeout": "60",
> > > > "mds_session_autoclose": "300",
> > > > "mds_reconnect_timeout": "45",
> > > > "mds_tick_interval": "5",
> > > > "mds_dirstat_min_interval": "1",
> > > > "mds_scatter_nudge_interval": "5",
> > > > "mds_client_prealloc_inos": "1000",
> > > > "mds_early_reply": "true",
> > > > "mds_use_tmap": "true",
> > > > "mds_default_dir_hash": "2",
> > > > "mds_log": "true",
> > > > "mds_log_skip_corrupt_events": "false",
> > > > "mds_log_max_events": "-1",
> > > > "mds_log_max_segments": "30",
> > > > "mds_log_max_expiring": "20",
> > > > "mds_bal_sample_interval": "3",
> > > > "mds_bal_replicate_threshold": "8000",
> > > > "mds_bal_unreplicate_threshold": "0",
> > > > "mds_bal_frag": "false",
> > > > "mds_bal_split_size": "10000",
> > > > "mds_bal_split_rd": "25000",
> > > > "mds_bal_split_wr": "10000",
> > > > "mds_bal_split_bits": "3",
> > > > "mds_bal_merge_size": "50",
> > > > "mds_bal_merge_rd": "1000",
> > > > "mds_bal_merge_wr": "1000",
> > > > "mds_bal_interval": "10",
> > > > "mds_bal_fragment_interval": "5",
> > > > "mds_bal_idle_threshold": "0",
> > > > "mds_bal_max": "-1",
> > > > "mds_bal_max_until": "-1",
> > > > "mds_bal_mode": "0",
> > > > "mds_bal_min_rebalance": "0.1",
> > > > "mds_bal_min_start": "0.2",
> > > > "mds_bal_need_min": "0.8",
> > > > "mds_bal_need_max": "1.2",
> > > > "mds_bal_midchunk": "0.3",
> > > > "mds_bal_minchunk": "0.001",
> > > > "mds_bal_target_removal_min": "5",
> > > > "mds_bal_target_removal_max": "10",
> > > > "mds_replay_interval": "1",
> > > > "mds_shutdown_check": "0",
> > > > "mds_thrash_exports": "0",
> > > > "mds_thrash_fragments": "0",
> > > > "mds_dump_cache_on_map": "false",
> > > > "mds_dump_cache_after_rejoin": "false",
> > > > "mds_verify_scatter": "false",
> > > > "mds_debug_scatterstat": "false",
> > > > "mds_debug_frag": "false",
> > > > "mds_debug_auth_pins": "false",
> > > > "mds_debug_subtrees": "false",
> > > > "mds_kill_mdstable_at": "0",
> > > > "mds_kill_export_at": "0",
> > > > "mds_kill_import_at": "0",
> > > > "mds_kill_link_at": "0",
> > > > "mds_kill_rename_at": "0",
> > > > "mds_wipe_sessions": "false",
> > > > "mds_wipe_ino_prealloc": "false",
> > > > "mds_skip_ino": "0",
> > > > "max_mds": "1",
> > > > "mds_standby_for_name": "",
> > > > "mds_standby_for_rank": "-1",
> > > > "mds_standby_replay": "false",
> > > > "osd_auto_upgrade_tmap": "true",
> > > > "osd_tmapput_sets_uses_tmap": "false",
> > > > "osd_max_backfills": "10",
> > > > "osd_backfill_full_ratio": "0.85",
> > > > "osd_backfill_retry_interval": "10",
> > > > "osd_uuid": "00000000-0000-0000-0000-000000000000",
> > > > "osd_data": "\/srv\/ceph\/osd\/osd.24",
> > > > "osd_journal": "\/dev\/sda3",
> > > > "osd_journal_size": "5120",
> > > > "osd_max_write_size": "90",
> > > > "osd_max_pgls": "1024",
> > > > "osd_client_message_size_cap": "524288000",
> > > > "osd_pg_bits": "6",
> > > > "osd_pgp_bits": "6",
> > > > "osd_min_rep": "1",
> > > > "osd_max_rep": "10",
> > > > "osd_pool_default_crush_rule": "0",
> > > > "osd_pool_default_size": "2",
> > > > "osd_pool_default_min_size": "0",
> > > > "osd_pool_default_pg_num": "8",
> > > > "osd_pool_default_pgp_num": "8",
> > > > "osd_map_dedup": "true",
> > > > "osd_map_cache_size": "500",
> > > > "osd_map_message_max": "100",
> > > > "osd_op_threads": "2",
> > > > "osd_disk_threads": "1",
> > > > "osd_recovery_threads": "1",
> > > > "osd_recover_clone_overlap": "true",
> > > > "osd_backfill_scan_min": "64",
> > > > "osd_backfill_scan_max": "512",
> > > > "osd_op_thread_timeout": "30",
> > > > "osd_recovery_thread_timeout": "30",
> > > > "osd_snap_trim_thread_timeout": "3600",
> > > > "osd_scrub_thread_timeout": "60",
> > > > "osd_scrub_finalize_thread_timeout": "600",
> > > > "osd_remove_thread_timeout": "3600",
> > > > "osd_command_thread_timeout": "600",
> > > > "osd_age": "0.8",
> > > > "osd_age_time": "0",
> > > > "osd_heartbeat_addr": ":\/0",
> > > > "osd_heartbeat_interval": "6",
> > > > "osd_heartbeat_grace": "20",
> > > > "osd_mon_heartbeat_interval": "30",
> > > > "osd_mon_report_interval_max": "120",
> > > > "osd_mon_report_interval_min": "5",
> > > > "osd_mon_ack_timeout": "30",
> > > > "osd_min_down_reporters": "1",
> > > > "osd_min_down_reports": "3",
> > > > "osd_default_data_pool_replay_window": "45",
> > > > "osd_preserve_trimmed_log": "false",
> > > > "osd_auto_mark_unfound_lost": "false",
> > > > "osd_recovery_delay_start": "0",
> > > > "osd_recovery_max_active": "50",
> > > > "osd_recovery_max_chunk": "8388608",
> > > > "osd_recovery_forget_lost_objects": "false",
> > > > "osd_max_scrubs": "1",
> > > > "osd_scrub_load_threshold": "0.5",
> > > > "osd_scrub_min_interval": "300",
> > > > "osd_scrub_max_interval": "86400",
> > > > "osd_deep_scrub_interval": "604800",
> > > > "osd_deep_scrub_stride": "524288",
> > > > "osd_auto_weight": "false",
> > > > "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
> > > > "osd_check_for_log_corruption": "false",
> > > > "osd_use_stale_snap": "false",
> > > > "osd_rollback_to_cluster_snap": "",
> > > > "osd_default_notify_timeout": "30",
> > > > "osd_kill_backfill_at": "0",
> > > > "osd_min_pg_log_entries": "1000",
> > > > "osd_op_complaint_time": "30",
> > > > "osd_command_max_records": "256",
> > > > "osd_op_log_threshold": "5",
> > > > "osd_verify_sparse_read_holes": "false",
> > > > "osd_debug_drop_ping_probability": "0",
> > > > "osd_debug_drop_ping_duration": "0",
> > > > "osd_debug_drop_pg_create_probability": "0",
> > > > "osd_debug_drop_pg_create_duration": "1",
> > > > "osd_debug_drop_op_probability": "0",
> > > > "osd_op_history_size": "20",
> > > > "osd_op_history_duration": "600",
> > > > "osd_target_transaction_size": "300",
> > > > "osd_client_op_priority": "63",
> > > > "osd_recovery_op_priority": "10",
> > > > "filestore": "false",
> > > > "filestore_index_retry_probability": "0",
> > > > "filestore_debug_omap_check": "false",
> > > > "filestore_xattr_use_omap": "false",
> > > > "filestore_max_inline_xattr_size": "512",
> > > > "filestore_max_inline_xattrs": "2",
> > > > "filestore_max_sync_interval": "5",
> > > > "filestore_min_sync_interval": "0.01",
> > > > "filestore_btrfs_snap": "true",
> > > > "filestore_btrfs_clone_range": "true",
> > > > "filestore_fsync_flushes_journal_data": "false",
> > > > "filestore_fiemap": "false",
> > > > "filestore_flusher": "true",
> > > > "filestore_flusher_max_fds": "512",
> > > > "filestore_flush_min": "65536",
> > > > "filestore_sync_flush": "false",
> > > > "filestore_journal_parallel": "false",
> > > > "filestore_journal_writeahead": "false",
> > > > "filestore_journal_trailing": "false",
> > > > "filestore_queue_max_ops": "500",
> > > > "filestore_queue_max_bytes": "104857600",
> > > > "filestore_queue_committing_max_ops": "500",
> > > > "filestore_queue_committing_max_bytes": "104857600",
> > > > "filestore_op_threads": "2",
> > > > "filestore_op_thread_timeout": "60",
> > > > "filestore_op_thread_suicide_timeout": "180",
> > > > "filestore_commit_timeout": "600",
> > > > "filestore_fiemap_threshold": "4096",
> > > > "filestore_merge_threshold": "10",
> > > > "filestore_split_multiple": "2",
> > > > "filestore_update_to": "1000",
> > > > "filestore_blackhole": "false",
> > > > "filestore_dump_file": "",
> > > > "filestore_kill_at": "0",
> > > > "filestore_fail_eio": "true",
> > > > "journal_dio": "true",
> > > > "journal_aio": "false",
> > > > "journal_block_align": "true",
> > > > "journal_max_write_bytes": "10485760",
> > > > "journal_max_write_entries": "100",
> > > > "journal_queue_max_ops": "500",
> > > > "journal_queue_max_bytes": "104857600",
> > > > "journal_align_min_size": "65536",
> > > > "journal_replay_from": "0",
> > > > "journal_zero_on_create": "false",
> > > > "rbd_cache": "false",
> > > > "rbd_cache_size": "33554432",
> > > > "rbd_cache_max_dirty": "25165824",
> > > > "rbd_cache_target_dirty": "16777216",
> > > > "rbd_cache_max_dirty_age": "1",
> > > > "nss_db_path": "",
> > > > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
> > > > "rgw_enable_apis": "s3, swift, swift_auth, admin",
> > > > "rgw_cache_enabled": "true",
> > > > "rgw_cache_lru_size": "10000",
> > > > "rgw_socket_path": "",
> > > > "rgw_dns_name": "",
> > > > "rgw_swift_url": "",
> > > > "rgw_swift_url_prefix": "swift",
> > > > "rgw_swift_auth_url": "",
> > > > "rgw_swift_auth_entry": "auth",
> > > > "rgw_keystone_url": "",
> > > > "rgw_keystone_admin_token": "",
> > > > "rgw_keystone_accepted_roles": "Member, admin",
> > > > "rgw_keystone_token_cache_size": "10000",
> > > > "rgw_keystone_revocation_interval": "900",
> > > > "rgw_admin_entry": "admin",
> > > > "rgw_enforce_swift_acls": "true",
> > > > "rgw_swift_token_expiration": "86400",
> > > > "rgw_print_continue": "true",
> > > > "rgw_remote_addr_param": "REMOTE_ADDR",
> > > > "rgw_op_thread_timeout": "600",
> > > > "rgw_op_thread_suicide_timeout": "0",
> > > > "rgw_thread_pool_size": "100",
> > > > "rgw_num_control_oids": "8",
> > > > "rgw_cluster_root_pool": ".rgw.root",
> > > > "rgw_log_nonexistent_bucket": "false",
> > > > "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > > > "rgw_log_object_name_utc": "false",
> > > > "rgw_usage_max_shards": "32",
> > > > "rgw_usage_max_user_shards": "1",
> > > > "rgw_enable_ops_log": "true",
> > > > "rgw_enable_usage_log": "true",
> > > > "rgw_ops_log_rados": "true",
> > > > "rgw_ops_log_socket_path": "",
> > > > "rgw_ops_log_data_backlog": "5242880",
> > > > "rgw_usage_log_flush_threshold": "1024",
> > > > "rgw_usage_log_tick_interval": "30",
> > > > "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
> > > > "rgw_intent_log_object_name_utc": "false",
> > > > "rgw_init_timeout": "30",
> > > > "rgw_mime_types_file": "\/etc\/mime.types",
> > > > "rgw_gc_max_objs": "32",
> > > > "rgw_gc_obj_min_wait": "7200",
> > > > "rgw_gc_processor_max_time": "3600",
> > > > "rgw_gc_processor_period": "3600",
> > > > "rgw_s3_success_create_obj_status": "0",
> > > > "rgw_resolve_cname": "false",
> > > > "rgw_obj_stripe_size": "4194304",
> > > > "rgw_extended_http_attrs": "",
> > > > "mutex_perf_counter": "false",
> > > > "internal_safe_to_start_threads": "true"}
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Nelson [mailto:mark.nelson@inktank.com]
> > > > Sent: Thursday, 20 December 2012 11:28 PM
> > > > To: Matthew Anderson
> > > > Cc: ceph-devel@vger.kernel.org
> > > > Subject: Re: OSD's slow down to a crawl
> > > >
> > > > Hi Matt,
> > > >
> > > > Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
> > > > Sebastien Han made a nice little blog entry describing it:
> > > >
> > > > http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
> > > >
> > > > You can get a lot of information about the current state of things.
> > > >
> > > > Mark
> > > >
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > in the body of a message to majordomo@vger.kernel.org More
> > > majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> > >
> >
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: OSD's slow down to a crawl
2013-01-09 8:52 ` Matthew Anderson
@ 2013-01-09 14:09 ` Mark Nelson
0 siblings, 0 replies; 13+ messages in thread
From: Mark Nelson @ 2013-01-09 14:09 UTC (permalink / raw)
To: Matthew Anderson; +Cc: 'Sage Weil', ceph-devel@vger.kernel.org
On 01/09/2013 02:52 AM, Matthew Anderson wrote:
> Hi Sage,
>
> Sorry for the late follow up, I've been on a bit of a testing rampage and managed to somewhat sort the problem.
>
> Most of the problems appears to be from the 3.7.1 kernel. It seems to have a fairly big issue with its networking stack that was causing Ceph's network operations to hang. Moving back to a 3.6.8 kernel fixed this up. I don't know the full extent of the problem but it was reported on Phoronix briefly here - http://www.phoronix.com/scan.php?page=news_item&px=MTI2Nzc
>
> The second issue was BTRFS on both the 3.7 and 3.6.8 kernels. After running a long rados bench (10 minutes) on a fresh cluster it would often slow down significantly by going from 250MB/s down to a 50MB/s average. Latency also increased dramatically. Restarting the OSD's fixes the issue but after a while it slows right down again. In the end I re-formatted the cluster using XFS (and also EXT4 for benchmarks) and there wasn't a single issue. I had rados bench running for over 30 minutes from another machine and there wasn't a single issue.
Ah, too bad this is still happening. :( It's interesting though that
restarting the OSDs fixes it. That's not something I expected. Sounds
like I need to run some more tests again and see if I can get to the
bottom of it.
>
> At thisstage I need to start moving into production with XFS. My test cluster arrives in a few weeks so I should be able to come back to the BTRFS issue later on as it would be very handy to have compression working.
>
> Thanks again for your help
> -Matt
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@inktank.com]
> Sent: Saturday, 22 December 2012 12:02 AM
> To: Matthew Anderson
> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
> Subject: RE: OSD's slow down to a crawl
>
> On Fri, 21 Dec 2012, Matthew Anderson wrote:
>> Hi Sage,
>>
>> I've tried to reproduce the error again with logging on every OSD and
>> got the above. RADOS bench had stalled on a write request like the
>> last time and the attached log is the grep'd OSD log (# cat osd.25.log
>> | grep client.9501.0:744> freeze.log) . The OSD that stalled was 25,
>> pg map is below -
>>
>> # ceph pg map 6.5d83495b
>> osdmap e3775 pg 6.5d83495b (6.95b) -> up [25,31] acting [25,31]
>>
>> I hope that's what you were after, if not just let me know
>
> We're getting closer. The osd tried to send the reply. Can you reproduce with 'debug ms = 20' on the osds too, and on the client side do soemthing like
>
> rados --debug-ms 20 --debug-objecter 20 --log-file /tmp/foo ...
>
> Thanks!
> sage
>
>
>>
>> Thanks again
>> -Matt
>>
>>
>> -----Original Message-----
>> From: Sage Weil [mailto:sage@inktank.com]
>> Sent: Friday, 21 December 2012 1:14 AM
>> To: Matthew Anderson
>> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
>> Subject: RE: OSD's slow down to a crawl
>>
>> On Thu, 20 Dec 2012, Matthew Anderson wrote:
>>> Hi Sage,
>>>
>>> Logs are attached. I took the osd logs from osd.24 as this is the
>>> first osd in my SSD pool I've been testing with previously.
>>>
>>> The 4MB bench I was able to reproduce the fault by restarting my rbd
>>> export which stalled after a few percent complete. When I ran the
>>> 4MB bench it stalled early on and never received a response back
>>> from the OSD and I terminated it after 60 seconds or so. I wasn't
>>> able to reproduce the fault using the 4kb io size. The 4kb log
>>> should show rados bench completing normally at a respectable speed of about 1MB/s.
>>
>> Let's drill into the hang.. up until that point things look okay.
>>
>> 2012-12-21 00:51:26.033622 7f6f3c042760 1 -- 172.16.0.13:0/1023886
>> --> 172.16.0.13:6813/22233 -- osd_op(client.9503.0:185
>> benchmark_data_KVM04_23886_object184 [write 0~4194304] 6.3ca4346e) v4
>> -- ?+0 0x171ea50 con 0x171a7e0
>>
>> Do you have a log for that OSD so we can see what happened there? It
>> may also be that the replicated write is hung. If you do
>>
>> ceph pg map 6.3ca4346e
>>
>> you can see all OSDs storing that PG. And/or you can grep for
>> client.9503.0:185 in 172.16.0.13:6813/22233's log and see whether the sub_op was sent.
>>
>> Thanks!
>> sage
>>
>>
>>>
>>> Thanks
>>> -Matt
>>>
>>> -----Original Message-----
>>> From: ceph-devel-owner@vger.kernel.org
>>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
>>> Sent: Friday, 21 December 2012 12:30 AM
>>> To: Matthew Anderson
>>> Cc: 'Mark Nelson'; ceph-devel@vger.kernel.org
>>> Subject: RE: OSD's slow down to a crawl
>>>
>>> Can you do a similar test, but with full logging on?
>>>
>>> ceph tell osd.0 injectargs '--debug-ms 1 --debug-filestore 20
>>> --debug-osd
>>> 20 --debug-journal 20'
>>> rados -p ssd bench 30 write -b 4096 -t 1 --log-file /tmp/foo
>>> --debug-ms 1
>>>
>>> That will be a single IO in flight at a time and very easy to trace through the logs. If you can post the resulting log file (/tmp/foo and from osd.0), that would be awesome.
>>>
>>> Thanks!
>>> sage
>>>
>>>
>>>
>>> On Thu, 20 Dec 2012, Matthew Anderson wrote:
>>>
>>>> # rados bench 60 write -t 256 -p ssd Maintaining 256 concurrent
>>>> writes of 4194304 bytes for at least 60 seconds.
>>>> Object prefix: benchmark_data_KVM03_12985
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 0 0 0 0 0 0 - 0
>>>> 1 45 45 0 0 0 - 0
>>>> 2 71 71 0 0 0 - 0
>>>> 3 90 90 0 0 0 - 0
>>>> 4 125 125 0 0 0 - 0
>>>> 5 136 136 0 0 0 - 0
>>>> 6 144 144 0 0 0 - 0
>>>> 7 175 175 0 0 0 - 0
>>>> 8 201 201 0 0 0 - 0
>>>> 9 217 217 0 0 0 - 0
>>>> 10 228 228 0 0 0 - 0
>>>> 11 251 251 0 0 0 - 0
>>>> 12 255 266 11 3.66581 3.66667 11.7027 11.6013
>>>> 13 255 271 16 4.92194 20 12.9795 11.9979
>>>> 14 255 291 36 10.2833 80 13.1795 12.6478
>>>> 15 255 301 46 12.2639 40 13.9424 12.9123
>>>> 16 255 303 48 11.9973 8 14.0277 12.9579
>>>> 17 255 321 66 15.526 72 15.1408 13.4616
>>>> 18 255 345 90 19.9956 96 15.5673 13.9937
>>>> 19 255 349 94 19.7851 16 15.707 14.0635
>>>> 2012-12-21 00:06:37.198982min lat: 11.4904 max lat: 16.4053 avg lat: 14.3121
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 20 255 361 106 21.1953 48 16.4053 14.3121
>>>> 21 255 367 112 21.3285 24 17.4263 14.4795
>>>> 22 255 370 115 20.9043 12 17.3579 14.5544
>>>> 23 255 387 132 22.9513 68 18.3185 15.0867
>>>> 24 255 394 139 23.1614 28 18.2868 15.2504
>>>> 25 255 404 149 23.8346 40 18.741 15.5048
>>>> 26 255 410 155 23.8407 24 19.032 15.6358
>>>> 27 255 410 155 22.9577 0 - 15.6358
>>>> 28 255 410 155 22.1378 0 - 15.6358
>>>> 29 255 411 156 21.5124 1.33333 22.6927 15.6811
>>>> 30 255 431 176 23.4614 80 22.6577 16.4822
>>>> 31 255 447 192 24.7687 64 23.2345 17.0563
>>>> 32 255 464 209 26.1192 68 23.3788 17.5859
>>>> 33 255 485 230 27.8726 84 22.8924 18.1122
>>>> 34 255 497 242 28.4642 48 23.563 18.3693
>>>> 35 255 508 253 28.9078 44 23.0516 18.5935
>>>> 36 255 514 259 28.7713 24 24.4415 18.7278
>>>> 37 255 530 275 29.723 64 23.847 19.0424
>>>> 38 255 541 286 30.0985 44 24.5757 19.2427
>>>> 39 255 544 289 29.6344 12 24.6802 19.2984
>>>> 2012-12-21 00:06:57.203473min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 40 255 544 289 28.8935 0 - 19.2984
>>>> 41 255 544 289 28.1888 0 - 19.2984
>>>> 42 255 544 289 27.5176 0 - 19.2984
>>>> 43 255 544 289 26.8776 0 - 19.2984
>>>> 44 255 544 289 26.2668 0 - 19.2984
>>>> 45 255 544 289 25.6831 0 - 19.2984
>>>> 46 255 544 289 25.1248 0 - 19.2984
>>>> 47 255 544 289 24.5902 0 - 19.2984
>>>> 48 255 544 289 24.0779 0 - 19.2984
>>>> 49 255 544 289 23.5865 0 - 19.2984
>>>> 50 255 544 289 23.1148 0 - 19.2984
>>>> 51 255 544 289 22.6616 0 - 19.2984
>>>> 52 255 544 289 22.2258 0 - 19.2984
>>>> 53 255 544 289 21.8065 0 - 19.2984
>>>> 54 255 544 289 21.4026 0 - 19.2984
>>>> 55 255 544 289 21.0135 0 - 19.2984
>>>> 56 255 544 289 20.6383 0 - 19.2984
>>>> 57 255 544 289 20.2762 0 - 19.2984
>>>> 58 255 544 289 19.9266 0 - 19.2984
>>>> 59 255 544 289 19.5889 0 - 19.2984
>>>> 2012-12-21 00:07:17.207782min lat: 11.4904 max lat: 24.6802 avg lat: 19.2984
>>>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>>>> 60 255 544 289 19.2624 0 - 19.2984
>>>> 61 255 544 289 18.9466 0 - 19.2984
>>>> 62 255 544 289 18.641 0 - 19.2984
>>>> 63 255 544 289 18.3452 0 - 19.2984
>>>> 64 255 544 289 18.0585 0 - 19.2984
>>>> 65 255 544 289 17.7807 0 - 19.2984
>>>> 66 255 544 289 17.5113 0 - 19.2984
>>>> 67 133 545 412 24.5916 17.5714 37.3574 27.4834
>>>> Total time run: 68.000740
>>>> Total writes made: 545
>>>> Write size: 4194304
>>>> Bandwidth (MB/sec): 32.058
>>>>
>>>> Stddev Bandwidth: 26.8837
>>>> Max bandwidth (MB/sec): 96
>>>> Min bandwidth (MB/sec): 0
>>>> Average Latency: 29.2801
>>>> Stddev Latency: 11.9768
>>>> Max latency: 52.6277
>>>> Min latency: 11.4904
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ----Original Message-----
>>>> From: Mark Nelson [mailto:mark.nelson@inktank.com]
>>>> Sent: Thursday, 20 December 2012 11:59 PM
>>>> To: Matthew Anderson
>>>> Cc: ceph-devel@vger.kernel.org
>>>> Subject: Re: OSD's slow down to a crawl
>>>>
>>>> Out of curiosity, if you fire up a rados bench instance on one of the nodes with say, 256 concurrent writes, do any of the writes complete?
>>>>
>>>> Mark
>>>>
>>>> On 12/20/2012 09:51 AM, Matthew Anderson wrote:
>>>>> Hi Mark,
>>>>>
>>>>> Thanks for the quick reply. The log was 380MB in total so I just posted last 200 lines hoping that a trained eye might be able to spot something. I'm happy to put the file up on a web server if anyone wants to review it in full.
>>>>>
>>>>> It doesn't appear to be memory or CPU related. In it's hung state (both read and write are not responding at the moment, I have a VM hung trying to shutdown and an rbd stuck exporting ) all of the OSD's sit around 0.3% CPU usage, ~300MB of ram and no IO wait. The server has roughly 40GB RAM free and is easily 90% idle. Iotop shows no IO coming from any of the Ceph-OSD's.
>>>>>
>>>>> The admin socket showed no ops in flight but the performance dump, perhistoric ops and config are below. I did notice that the age of some of the ops was quite high considering things were actively trying to write to the cluster.
>>>>> # ceph --admin-daemon ./ceph-osd.24.asok perf dump
>>>>>
>>>>> {"filestore":{"journal_queue_max_ops":500,"journal_queue_ops":0,
>>>>> "j
>>>>> ou
>>>>> rn
>>>>> al_ops":20868,"journal_queue_max_bytes":104857600,"journal_queue
>>>>> _b
>>>>> yt
>>>>> es
>>>>> ":0,"journal_bytes":1710397238,"journal_latency":{"avgcount":20868,"
>>>>> su
>>>>> m":185.268022000},"journal_wr":13354,"journal_wr_bytes":{"avgcount":
>>>>> 13
>>>>> 354,"sum":1770647552},"op_queue_max_ops":500,"op_queue_ops":0,"ops":
>>>>> 20
>>>>> 868,"op_queue_max_bytes":104857600,"op_queue_bytes":0,"bytes":17
>>>>> 10
>>>>> 23
>>>>> 25
>>>>> 83,"apply_latency":{"avgcount":20868,"sum":508.896737000},"committing"
>>>>> :0,"commitcycle":339,"commitcycle_interval":{"avgcount":339,"sum":
>>>>> 17
>>>>> 08
>>>>> .284725000},"commitcycle_latency":{"avgcount":339,"sum":13.24185
>>>>> 00
>>>>> 00
>>>>> },
>>>>> "journal_full":0},"mutex-FileJournal::completions_lock":{"wait":{"
>>>>> av
>>>>> gc
>>>>> ount":0,"sum":0.000000000}},"mutex-FileJournal::finisher_lock":{"wait"
>>>>> :{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write_lock"
>>>>> :{
>>>>> "w
>>>>> ait":{"avgcount":0,"sum":0.000000000}},"mutex-FileJournal::write
>>>>> q_
>>>>> lo
>>>>> ck
>>>>> ":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::
>>>>> apply_lo
>>>> ck":{"wai
>>>> t":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::ApplyManager::com
>>>> _l
>>>> oc
>>>> k":{"wait":{"avgcount":0,"sum":0.000000000}},"mutex-JOS::SubmitMan
>>>> ag
>>>> er
>>>> ::lock":{"wait":{"avgcount":0,"sum":0.000000000}},"osd":{"opq":0,"
>>>> op
>>>> _w
>>>> ip":0,"op":31753,"op_in_bytes":797806224,"op_out_bytes":1728414048,"
>>>> op
>>>> _latency":{"avgcount":31753,"sum":82.183373000},"op_r":24945,"op_r
>>>> _o
>>>> ut
>>>> _bytes":1728414048,"op_r_latency":{"avgcount":24945,"sum":16.83368
>>>> 90
>>>> 00
>>>> },"op_w":6808,"op_w_in_bytes":797806224,"op_w_rlat":{"avgcount":6808,"
>>>> sum":38.223006000},"op_w_latency":{"avgcount":6808,"sum":65.349684
>>>> 00
>>>> 0}
>>>> ,"op_rw":0,"op_rw_in_bytes":0,"op_rw_out_bytes":0,"op_rw_rlat":{"a
>>>> vg
>>>> co
>>>> unt":0,"sum":0.000000000},"op_rw_latency":{"avgcount":0,"sum":0.00
>>>> 00
>>>> 00
>>>> 000},"subop":5700,"subop_in_bytes":813224156,"subop_latency":{"avg
>>>> co
>>>> un
>>>> t":5700,"sum":26.877085000},"subop_w":0,"subop_w_in_bytes":813224156,"
>>>> subop_w_latency":{"avgcount":5700,"sum":26.877085000},"subop_pull":0,"
>>>> subop_pull_latency":{"avgcount":0,"sum":0.000000000},"subop_push":
>>>> 0,
>>>> "s
>>>> ubop_push_
>>>> in_bytes"
>>>> :0,"subop_push_latency":{"avgcount":0,"sum":0.000000000},"pull":0,
>>>> "p
>>>> us
>>>> h":0,"push_out_bytes":0,"push_in":0,"push_in_bytes":0,"recovery_ops"
>>>> :0
>>>> ,"loadavg":359,"buffer_bytes":0,"numpg":2486,"numpg_primary":1214,
>>>> "n
>>>> um
>>>> pg_replica":1272,"numpg_stray":0,"heartbeat_to_peers":4,"heartbeat
>>>> _f
>>>> ro
>>>> m_peers":0,"map_messages":44,"map_message_epochs":184,"map_message
>>>> _e
>>>> po
>>>> ch_dups":174},"throttle-filestore_bytes":{"val":0,"max":104857600,
>>>> "g
>>>> et
>>>> ":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":
>>>> 20
>>>> 868,"take_sum":1710397238,"put":13354,"put_sum":1710397238,"wait":{"
>>>> av
>>>> gcount":0,"sum":0.000000000}},"throttle-filestore_ops":{"val":0,"max":
>>>> 500,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"
>>>> take":20868,"take_sum":20868,"put":13354,"put_sum":20868,"wait":{"
>>>> av
>>>> gc
>>>> ount":0,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-client":
>>>> {"val":0,"max":104857600,"get":31657,"get_sum":795546573,"get_or_f
>>>> ai
>>>> l_
>>>> fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":31657,
>>>> "p
>>>> ut
>>>> _sum":7955
>>>> 46573,"wa
>>>> it":{"avgcount":0,"sum":0.000000000}},"throttle-msgr_dispatch_thro
>>>> tt
>>>> le
>>>> r-cluster":{"val":0,"max":104857600,"get":18913,"get_sum":82801129
>>>> 3,
>>>> "g
>>>> et_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put"
>>>> :18913,"put_sum":828011293,"wait":{"avgcount":0,"sum":0.000000000}
>>>> },
>>>> "t
>>>> hrottle-msgr_dispatch_throttler-hbclient":{"val":0,"max":104857600,"
>>>> ge
>>>> t":4260,"get_sum":200220,"get_or_fail_fail":0,"get_or_fail_success":
>>>> 0,
>>>> "take":0,"take_sum":0,"put":4260,"put_sum":200220,"wait":{"avgcount"
>>>> :0
>>>> ,"sum":0.000000000}},"throttle-msgr_dispatch_throttler-hbserver":{
>>>> "v
>>>> al
>>>> ":0,"max":104857600,"get":4236,"get_sum":199092,"get_or_fail_fail":0,"
>>>> get_or_fail_success":0,"take":0,"take_sum":0,"put":4236,"put_sum":
>>>> 19
>>>> 90
>>>> 92,"wait":{"avgcount":0,"sum":0.000000000}},"throttle-osd_client_b
>>>> yt
>>>> es
>>>> ":{"val":0,"max":524288000,"get":31308,"get_sum":795353720,"get_or
>>>> _f
>>>> ai
>>>> l_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":3767
>>>> 1, "p ut_sum":795353720,"wait":{"avgcount":0,"sum":0.000000000}}}
>>>>>
>>>>> # ceph --admin-daemon ./ceph-osd.24.asok dump_historic_ops
>>>>>
>>>>> { "description": "osd_op(client.7636.0:101498 rb.0.122c.6b8b4567.000000000357 [write 3600384~4096] 6.42463fe3)",
>>>>> "received_at": "2012-12-20 23:34:42.012131",
>>>>> "age": "75.428051",
>>>>> "duration": "0.009988",
>>>>> "flag_point": "waiting for sub ops",
>>>>> "client_info": { "client": "client.7636",
>>>>> "tid": 101498},
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:34:42.012317",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:34:42.012425",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:34:42.012678",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:34:42.012720",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:34:42.012815",
>>>>> "event": "sub_op_sent"},
>>>>> { "time": "2012-12-20 23:34:42.012958",
>>>>> "event": "commit_queued_for_journal_write"},
>>>>> { "time": "2012-12-20 23:34:42.013033",
>>>>> "event": "write_thread_in_journal_buffer"},
>>>>> { "time": "2012-12-20 23:34:42.013745",
>>>>> "event": "journaled_completion_queued"},
>>>>> { "time": "2012-12-20 23:34:42.013803",
>>>>> "event": "op_commit"},
>>>>> { "time": "2012-12-20 23:34:42.017414",
>>>>> "event": "sub_op_commit_rec"},
>>>>> { "time": "2012-12-20 23:34:42.022066",
>>>>> "event": "op_applied"},
>>>>> { "time": "2012-12-20 23:34:42.022119",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_sub_op_reply(client.7636.0:101498 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
>>>>> "received_at": "2012-12-20 23:34:42.017048",
>>>>> "age": "75.423134",
>>>>> "duration": "0.000411",
>>>>> "flag_point": "started",
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:34:42.017218",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:34:42.017363",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:34:42.017401",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:34:42.017459",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_op(client.7636.0:101502 rb.0.122c.6b8b4567.000000000064 [write 1512448~61440] 6.a241e34d)",
>>>>> "received_at": "2012-12-20 23:34:49.851021",
>>>>> "age": "67.589161",
>>>>> "duration": "0.007114",
>>>>> "flag_point": "waiting for sub ops",
>>>>> "client_info": { "client": "client.7636",
>>>>> "tid": 101502},
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:34:49.851276",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:34:49.851429",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:34:49.851616",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:34:49.851648",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:34:49.851712",
>>>>> "event": "sub_op_sent"},
>>>>> { "time": "2012-12-20 23:34:49.851770",
>>>>> "event": "commit_queued_for_journal_write"},
>>>>> { "time": "2012-12-20 23:34:49.851915",
>>>>> "event": "write_thread_in_journal_buffer"},
>>>>> { "time": "2012-12-20 23:34:49.853956",
>>>>> "event": "journaled_completion_queued"},
>>>>> { "time": "2012-12-20 23:34:49.853992",
>>>>> "event": "op_commit"},
>>>>> { "time": "2012-12-20 23:34:49.857016",
>>>>> "event": "sub_op_commit_rec"},
>>>>> { "time": "2012-12-20 23:34:49.858077",
>>>>> "event": "op_applied"},
>>>>> { "time": "2012-12-20 23:34:49.858135",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_sub_op_reply(client.7636.0:101502 6.234d a241e34d\/rb.0.122c.6b8b4567.000000000064\/head\/\/6 [] ondisk, result = 0)",
>>>>> "received_at": "2012-12-20 23:34:49.856762",
>>>>> "age": "67.583420",
>>>>> "duration": "0.000308",
>>>>> "flag_point": "started",
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:34:49.856869",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:34:49.856969",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:34:49.857001",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:34:49.857070",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_op(client.7636.0:101504 rb.0.122c.6b8b4567.00000000012c [write 1658880~2048] 6.92d2c38c)",
>>>>> "received_at": "2012-12-20 23:35:34.351518",
>>>>> "age": "23.088664",
>>>>> "duration": "0.009430",
>>>>> "flag_point": "waiting for sub ops",
>>>>> "client_info": { "client": "client.7636",
>>>>> "tid": 101504},
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:35:34.351714",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:35:34.351811",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:35:34.352030",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:35:34.352071",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:35:34.352146",
>>>>> "event": "sub_op_sent"},
>>>>> { "time": "2012-12-20 23:35:34.352213",
>>>>> "event": "commit_queued_for_journal_write"},
>>>>> { "time": "2012-12-20 23:35:34.352287",
>>>>> "event": "write_thread_in_journal_buffer"},
>>>>> { "time": "2012-12-20 23:35:34.355390",
>>>>> "event": "journaled_completion_queued"},
>>>>> { "time": "2012-12-20 23:35:34.355475",
>>>>> "event": "op_commit"},
>>>>> { "time": "2012-12-20 23:35:34.356661",
>>>>> "event": "sub_op_commit_rec"},
>>>>> { "time": "2012-12-20 23:35:34.360894",
>>>>> "event": "op_applied"},
>>>>> { "time": "2012-12-20 23:35:34.360948",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_sub_op_reply(client.7636.0:101504 6.38c 92d2c38c\/rb.0.122c.6b8b4567.00000000012c\/head\/\/6 [] ondisk, result = 0)",
>>>>> "received_at": "2012-12-20 23:35:34.356286",
>>>>> "age": "23.083896",
>>>>> "duration": "0.000421",
>>>>> "flag_point": "started",
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:35:34.356489",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:35:34.356598",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:35:34.356640",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:35:34.356707",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_op(client.7636.0:101512 rb.0.122c.6b8b4567.000000000357 [write 3731456~4096] 6.42463fe3)",
>>>>> "received_at": "2012-12-20 23:35:42.133103",
>>>>> "age": "15.307079",
>>>>> "duration": "0.008357",
>>>>> "flag_point": "waiting for sub ops",
>>>>> "client_info": { "client": "client.7636",
>>>>> "tid": 101512},
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:35:42.133315",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:35:42.133427",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:35:42.133684",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:35:42.133723",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:35:42.133796",
>>>>> "event": "sub_op_sent"},
>>>>> { "time": "2012-12-20 23:35:42.133908",
>>>>> "event": "commit_queued_for_journal_write"},
>>>>> { "time": "2012-12-20 23:35:42.133995",
>>>>> "event": "write_thread_in_journal_buffer"},
>>>>> { "time": "2012-12-20 23:35:42.136965",
>>>>> "event": "journaled_completion_queued"},
>>>>> { "time": "2012-12-20 23:35:42.137025",
>>>>> "event": "op_commit"},
>>>>> { "time": "2012-12-20 23:35:42.138456",
>>>>> "event": "sub_op_commit_rec"},
>>>>> { "time": "2012-12-20 23:35:42.141408",
>>>>> "event": "op_applied"},
>>>>> { "time": "2012-12-20 23:35:42.141460",
>>>>> "event": "done"}]},
>>>>> { "description": "osd_sub_op_reply(client.7636.0:101512 6.1fe3 42463fe3\/rb.0.122c.6b8b4567.000000000357\/head\/\/6 [] ondisk, result = 0)",
>>>>> "received_at": "2012-12-20 23:35:42.138087",
>>>>> "age": "15.302095",
>>>>> "duration": "0.000410",
>>>>> "flag_point": "started",
>>>>> "events": [
>>>>> { "time": "2012-12-20 23:35:42.138248",
>>>>> "event": "waiting_for_osdmap"},
>>>>> { "time": "2012-12-20 23:35:42.138363",
>>>>> "event": "reached_pg"},
>>>>> { "time": "2012-12-20 23:35:42.138443",
>>>>> "event": "started"},
>>>>> { "time": "2012-12-20 23:35:42.138497",
>>>>> "event": "done"}]}]}
>>>>>
>>>>> # ceph --admin-daemon ./ceph-osd.24.asok config show { "name":
>>>>> "osd.24",
>>>>> "cluster": "ceph",
>>>>> "none": "0\/5",
>>>>> "lockdep": "0\/1",
>>>>> "context": "0\/1",
>>>>> "crush": "1\/1",
>>>>> "mds": "1\/5",
>>>>> "mds_balancer": "1\/5",
>>>>> "mds_locker": "1\/5",
>>>>> "mds_log": "1\/5",
>>>>> "mds_log_expire": "1\/5",
>>>>> "mds_migrator": "1\/5",
>>>>> "buffer": "0\/1",
>>>>> "timer": "0\/1",
>>>>> "filer": "0\/1",
>>>>> "striper": "0\/1",
>>>>> "objecter": "0\/1",
>>>>> "rados": "0\/5",
>>>>> "rbd": "0\/5",
>>>>> "journaler": "0\/5",
>>>>> "objectcacher": "0\/5",
>>>>> "client": "0\/5",
>>>>> "osd": "0\/5",
>>>>> "optracker": "0\/5",
>>>>> "objclass": "0\/5",
>>>>> "filestore": "1\/3",
>>>>> "journal": "1\/3",
>>>>> "ms": "0\/5",
>>>>> "mon": "1\/5",
>>>>> "monc": "0\/10",
>>>>> "paxos": "0\/5",
>>>>> "tp": "0\/5",
>>>>> "auth": "1\/5",
>>>>> "crypto": "1\/5",
>>>>> "finisher": "1\/1",
>>>>> "heartbeatmap": "1\/5",
>>>>> "perfcounter": "1\/5",
>>>>> "rgw": "1\/5",
>>>>> "hadoop": "1\/5",
>>>>> "javaclient": "1\/5",
>>>>> "asok": "1\/5",
>>>>> "throttle": "1\/1",
>>>>> "host": "KVM04",
>>>>> "fsid": "00000000-0000-0000-0000-000000000000",
>>>>> "public_addr": ":\/0",
>>>>> "cluster_addr": ":\/0",
>>>>> "public_network": "",
>>>>> "cluster_network": "",
>>>>> "num_client": "1",
>>>>> "monmap": "",
>>>>> "mon_host": "",
>>>>> "lockdep": "false",
>>>>> "admin_socket": "\/var\/run\/ceph\/ceph-osd.24.asok",
>>>>> "daemonize": "true",
>>>>> "pid_file": "\/var\/run\/ceph\/osd.24.pid",
>>>>> "chdir": "\/",
>>>>> "max_open_files": "0",
>>>>> "fatal_signal_handlers": "true",
>>>>> "log_file": "\/var\/log\/ceph\/osd.24.log",
>>>>> "log_max_new": "1000",
>>>>> "log_max_recent": "100000",
>>>>> "log_to_stderr": "false",
>>>>> "err_to_stderr": "true",
>>>>> "log_to_syslog": "false",
>>>>> "err_to_syslog": "false",
>>>>> "log_flush_on_exit": "true",
>>>>> "clog_to_monitors": "true",
>>>>> "clog_to_syslog": "false",
>>>>> "mon_cluster_log_to_syslog": "false",
>>>>> "mon_cluster_log_file": "\/var\/log\/ceph\/ceph.log",
>>>>> "key": "",
>>>>> "keyfile": "",
>>>>> "keyring": "\/srv\/ceph\/osd\/osd.24\/keyring",
>>>>> "heartbeat_interval": "5",
>>>>> "heartbeat_file": "",
>>>>> "perf": "true",
>>>>> "ms_tcp_nodelay": "true",
>>>>> "ms_initial_backoff": "0.2",
>>>>> "ms_max_backoff": "15",
>>>>> "ms_nocrc": "false",
>>>>> "ms_die_on_bad_msg": "false",
>>>>> "ms_dispatch_throttle_bytes": "104857600",
>>>>> "ms_bind_ipv6": "false",
>>>>> "ms_bind_port_min": "6800",
>>>>> "ms_bind_port_max": "7100",
>>>>> "ms_rwthread_stack_bytes": "1048576",
>>>>> "ms_tcp_read_timeout": "900",
>>>>> "ms_inject_socket_failures": "0",
>>>>> "ms_inject_delay_type": "",
>>>>> "ms_inject_delay_max": "1",
>>>>> "ms_inject_delay_probability": "0",
>>>>> "mon_data": "\/var\/lib\/ceph\/mon\/ceph-24",
>>>>> "mon_initial_members": "",
>>>>> "mon_sync_fs_threshold": "5",
>>>>> "mon_tick_interval": "5",
>>>>> "mon_subscribe_interval": "300",
>>>>> "mon_osd_laggy_halflife": "3600",
>>>>> "mon_osd_laggy_weight": "0.3",
>>>>> "mon_osd_adjust_heartbeat_grace": "true",
>>>>> "mon_osd_adjust_down_out_interval": "true",
>>>>> "mon_osd_auto_mark_in": "false",
>>>>> "mon_osd_auto_mark_auto_out_in": "true",
>>>>> "mon_osd_auto_mark_new_in": "true",
>>>>> "mon_osd_down_out_interval": "300",
>>>>> "mon_osd_min_up_ratio": "0.3",
>>>>> "mon_osd_min_in_ratio": "0.3",
>>>>> "mon_lease": "5",
>>>>> "mon_lease_renew_interval": "3",
>>>>> "mon_lease_ack_timeout": "10",
>>>>> "mon_clock_drift_allowed": "0.05",
>>>>> "mon_clock_drift_warn_backoff": "5",
>>>>> "mon_accept_timeout": "10",
>>>>> "mon_pg_create_interval": "30",
>>>>> "mon_pg_stuck_threshold": "300",
>>>>> "mon_osd_full_ratio": "0.95",
>>>>> "mon_osd_nearfull_ratio": "0.85",
>>>>> "mon_globalid_prealloc": "100",
>>>>> "mon_osd_report_timeout": "900",
>>>>> "mon_force_standby_active": "true",
>>>>> "mon_min_osdmap_epochs": "500",
>>>>> "mon_max_pgmap_epochs": "500",
>>>>> "mon_max_log_epochs": "500",
>>>>> "mon_max_osd": "10000",
>>>>> "mon_probe_timeout": "2",
>>>>> "mon_slurp_timeout": "10",
>>>>> "mon_slurp_bytes": "262144",
>>>>> "mon_client_bytes": "104857600",
>>>>> "mon_daemon_bytes": "419430400",
>>>>> "mon_max_log_entries_per_event": "4096",
>>>>> "paxos_max_join_drift": "10",
>>>>> "paxos_propose_interval": "1",
>>>>> "paxos_min_wait": "0.05",
>>>>> "clock_offset": "0",
>>>>> "auth_cluster_required": "none",
>>>>> "auth_service_required": "none",
>>>>> "auth_client_required": "none",
>>>>> "auth_supported": "none",
>>>>> "cephx_require_signatures": "false",
>>>>> "cephx_sign_messages": "true",
>>>>> "auth_mon_ticket_ttl": "43200",
>>>>> "auth_service_ticket_ttl": "3600",
>>>>> "mon_client_hunt_interval": "3",
>>>>> "mon_client_ping_interval": "10",
>>>>> "mon_client_max_log_entries_per_message": "1000",
>>>>> "mon_max_pool_pg_num": "65536",
>>>>> "client_cache_size": "16384",
>>>>> "client_cache_mid": "0.75",
>>>>> "client_use_random_mds": "false",
>>>>> "client_mount_timeout": "30",
>>>>> "client_tick_interval": "1",
>>>>> "client_trace": "",
>>>>> "client_readahead_min": "131072",
>>>>> "client_readahead_max_bytes": "0",
>>>>> "client_readahead_max_periods": "4",
>>>>> "client_snapdir": ".snap",
>>>>> "client_mountpoint": "\/",
>>>>> "client_notify_timeout": "10",
>>>>> "client_caps_release_delay": "5",
>>>>> "client_oc": "true",
>>>>> "client_oc_size": "209715200",
>>>>> "client_oc_max_dirty": "104857600",
>>>>> "client_oc_target_dirty": "8388608",
>>>>> "client_oc_max_dirty_age": "5",
>>>>> "client_oc_max_objects": "1000",
>>>>> "client_debug_force_sync_read": "false",
>>>>> "client_debug_inject_tick_delay": "0",
>>>>> "fuse_use_invalidate_cb": "false",
>>>>> "fuse_big_writes": "true",
>>>>> "fuse_debug": "false",
>>>>> "objecter_tick_interval": "5",
>>>>> "objecter_timeout": "10",
>>>>> "objecter_inflight_op_bytes": "104857600",
>>>>> "objecter_inflight_ops": "1024",
>>>>> "journaler_allow_split_entries": "true",
>>>>> "journaler_write_head_interval": "15",
>>>>> "journaler_prefetch_periods": "10",
>>>>> "journaler_prezero_periods": "5",
>>>>> "journaler_batch_interval": "0.001",
>>>>> "journaler_batch_max": "0",
>>>>> "mds_data": "\/var\/lib\/ceph\/mds\/ceph-24",
>>>>> "mds_max_file_size": "1099511627776",
>>>>> "mds_cache_size": "100000",
>>>>> "mds_cache_mid": "0.7",
>>>>> "mds_mem_max": "1048576",
>>>>> "mds_dir_commit_ratio": "0.5",
>>>>> "mds_dir_max_commit_size": "90",
>>>>> "mds_decay_halflife": "5",
>>>>> "mds_beacon_interval": "4",
>>>>> "mds_beacon_grace": "15",
>>>>> "mds_blacklist_interval": "1440",
>>>>> "mds_session_timeout": "60",
>>>>> "mds_session_autoclose": "300",
>>>>> "mds_reconnect_timeout": "45",
>>>>> "mds_tick_interval": "5",
>>>>> "mds_dirstat_min_interval": "1",
>>>>> "mds_scatter_nudge_interval": "5",
>>>>> "mds_client_prealloc_inos": "1000",
>>>>> "mds_early_reply": "true",
>>>>> "mds_use_tmap": "true",
>>>>> "mds_default_dir_hash": "2",
>>>>> "mds_log": "true",
>>>>> "mds_log_skip_corrupt_events": "false",
>>>>> "mds_log_max_events": "-1",
>>>>> "mds_log_max_segments": "30",
>>>>> "mds_log_max_expiring": "20",
>>>>> "mds_bal_sample_interval": "3",
>>>>> "mds_bal_replicate_threshold": "8000",
>>>>> "mds_bal_unreplicate_threshold": "0",
>>>>> "mds_bal_frag": "false",
>>>>> "mds_bal_split_size": "10000",
>>>>> "mds_bal_split_rd": "25000",
>>>>> "mds_bal_split_wr": "10000",
>>>>> "mds_bal_split_bits": "3",
>>>>> "mds_bal_merge_size": "50",
>>>>> "mds_bal_merge_rd": "1000",
>>>>> "mds_bal_merge_wr": "1000",
>>>>> "mds_bal_interval": "10",
>>>>> "mds_bal_fragment_interval": "5",
>>>>> "mds_bal_idle_threshold": "0",
>>>>> "mds_bal_max": "-1",
>>>>> "mds_bal_max_until": "-1",
>>>>> "mds_bal_mode": "0",
>>>>> "mds_bal_min_rebalance": "0.1",
>>>>> "mds_bal_min_start": "0.2",
>>>>> "mds_bal_need_min": "0.8",
>>>>> "mds_bal_need_max": "1.2",
>>>>> "mds_bal_midchunk": "0.3",
>>>>> "mds_bal_minchunk": "0.001",
>>>>> "mds_bal_target_removal_min": "5",
>>>>> "mds_bal_target_removal_max": "10",
>>>>> "mds_replay_interval": "1",
>>>>> "mds_shutdown_check": "0",
>>>>> "mds_thrash_exports": "0",
>>>>> "mds_thrash_fragments": "0",
>>>>> "mds_dump_cache_on_map": "false",
>>>>> "mds_dump_cache_after_rejoin": "false",
>>>>> "mds_verify_scatter": "false",
>>>>> "mds_debug_scatterstat": "false",
>>>>> "mds_debug_frag": "false",
>>>>> "mds_debug_auth_pins": "false",
>>>>> "mds_debug_subtrees": "false",
>>>>> "mds_kill_mdstable_at": "0",
>>>>> "mds_kill_export_at": "0",
>>>>> "mds_kill_import_at": "0",
>>>>> "mds_kill_link_at": "0",
>>>>> "mds_kill_rename_at": "0",
>>>>> "mds_wipe_sessions": "false",
>>>>> "mds_wipe_ino_prealloc": "false",
>>>>> "mds_skip_ino": "0",
>>>>> "max_mds": "1",
>>>>> "mds_standby_for_name": "",
>>>>> "mds_standby_for_rank": "-1",
>>>>> "mds_standby_replay": "false",
>>>>> "osd_auto_upgrade_tmap": "true",
>>>>> "osd_tmapput_sets_uses_tmap": "false",
>>>>> "osd_max_backfills": "10",
>>>>> "osd_backfill_full_ratio": "0.85",
>>>>> "osd_backfill_retry_interval": "10",
>>>>> "osd_uuid": "00000000-0000-0000-0000-000000000000",
>>>>> "osd_data": "\/srv\/ceph\/osd\/osd.24",
>>>>> "osd_journal": "\/dev\/sda3",
>>>>> "osd_journal_size": "5120",
>>>>> "osd_max_write_size": "90",
>>>>> "osd_max_pgls": "1024",
>>>>> "osd_client_message_size_cap": "524288000",
>>>>> "osd_pg_bits": "6",
>>>>> "osd_pgp_bits": "6",
>>>>> "osd_min_rep": "1",
>>>>> "osd_max_rep": "10",
>>>>> "osd_pool_default_crush_rule": "0",
>>>>> "osd_pool_default_size": "2",
>>>>> "osd_pool_default_min_size": "0",
>>>>> "osd_pool_default_pg_num": "8",
>>>>> "osd_pool_default_pgp_num": "8",
>>>>> "osd_map_dedup": "true",
>>>>> "osd_map_cache_size": "500",
>>>>> "osd_map_message_max": "100",
>>>>> "osd_op_threads": "2",
>>>>> "osd_disk_threads": "1",
>>>>> "osd_recovery_threads": "1",
>>>>> "osd_recover_clone_overlap": "true",
>>>>> "osd_backfill_scan_min": "64",
>>>>> "osd_backfill_scan_max": "512",
>>>>> "osd_op_thread_timeout": "30",
>>>>> "osd_recovery_thread_timeout": "30",
>>>>> "osd_snap_trim_thread_timeout": "3600",
>>>>> "osd_scrub_thread_timeout": "60",
>>>>> "osd_scrub_finalize_thread_timeout": "600",
>>>>> "osd_remove_thread_timeout": "3600",
>>>>> "osd_command_thread_timeout": "600",
>>>>> "osd_age": "0.8",
>>>>> "osd_age_time": "0",
>>>>> "osd_heartbeat_addr": ":\/0",
>>>>> "osd_heartbeat_interval": "6",
>>>>> "osd_heartbeat_grace": "20",
>>>>> "osd_mon_heartbeat_interval": "30",
>>>>> "osd_mon_report_interval_max": "120",
>>>>> "osd_mon_report_interval_min": "5",
>>>>> "osd_mon_ack_timeout": "30",
>>>>> "osd_min_down_reporters": "1",
>>>>> "osd_min_down_reports": "3",
>>>>> "osd_default_data_pool_replay_window": "45",
>>>>> "osd_preserve_trimmed_log": "false",
>>>>> "osd_auto_mark_unfound_lost": "false",
>>>>> "osd_recovery_delay_start": "0",
>>>>> "osd_recovery_max_active": "50",
>>>>> "osd_recovery_max_chunk": "8388608",
>>>>> "osd_recovery_forget_lost_objects": "false",
>>>>> "osd_max_scrubs": "1",
>>>>> "osd_scrub_load_threshold": "0.5",
>>>>> "osd_scrub_min_interval": "300",
>>>>> "osd_scrub_max_interval": "86400",
>>>>> "osd_deep_scrub_interval": "604800",
>>>>> "osd_deep_scrub_stride": "524288",
>>>>> "osd_auto_weight": "false",
>>>>> "osd_class_dir": "\/usr\/local\/lib\/rados-classes",
>>>>> "osd_check_for_log_corruption": "false",
>>>>> "osd_use_stale_snap": "false",
>>>>> "osd_rollback_to_cluster_snap": "",
>>>>> "osd_default_notify_timeout": "30",
>>>>> "osd_kill_backfill_at": "0",
>>>>> "osd_min_pg_log_entries": "1000",
>>>>> "osd_op_complaint_time": "30",
>>>>> "osd_command_max_records": "256",
>>>>> "osd_op_log_threshold": "5",
>>>>> "osd_verify_sparse_read_holes": "false",
>>>>> "osd_debug_drop_ping_probability": "0",
>>>>> "osd_debug_drop_ping_duration": "0",
>>>>> "osd_debug_drop_pg_create_probability": "0",
>>>>> "osd_debug_drop_pg_create_duration": "1",
>>>>> "osd_debug_drop_op_probability": "0",
>>>>> "osd_op_history_size": "20",
>>>>> "osd_op_history_duration": "600",
>>>>> "osd_target_transaction_size": "300",
>>>>> "osd_client_op_priority": "63",
>>>>> "osd_recovery_op_priority": "10",
>>>>> "filestore": "false",
>>>>> "filestore_index_retry_probability": "0",
>>>>> "filestore_debug_omap_check": "false",
>>>>> "filestore_xattr_use_omap": "false",
>>>>> "filestore_max_inline_xattr_size": "512",
>>>>> "filestore_max_inline_xattrs": "2",
>>>>> "filestore_max_sync_interval": "5",
>>>>> "filestore_min_sync_interval": "0.01",
>>>>> "filestore_btrfs_snap": "true",
>>>>> "filestore_btrfs_clone_range": "true",
>>>>> "filestore_fsync_flushes_journal_data": "false",
>>>>> "filestore_fiemap": "false",
>>>>> "filestore_flusher": "true",
>>>>> "filestore_flusher_max_fds": "512",
>>>>> "filestore_flush_min": "65536",
>>>>> "filestore_sync_flush": "false",
>>>>> "filestore_journal_parallel": "false",
>>>>> "filestore_journal_writeahead": "false",
>>>>> "filestore_journal_trailing": "false",
>>>>> "filestore_queue_max_ops": "500",
>>>>> "filestore_queue_max_bytes": "104857600",
>>>>> "filestore_queue_committing_max_ops": "500",
>>>>> "filestore_queue_committing_max_bytes": "104857600",
>>>>> "filestore_op_threads": "2",
>>>>> "filestore_op_thread_timeout": "60",
>>>>> "filestore_op_thread_suicide_timeout": "180",
>>>>> "filestore_commit_timeout": "600",
>>>>> "filestore_fiemap_threshold": "4096",
>>>>> "filestore_merge_threshold": "10",
>>>>> "filestore_split_multiple": "2",
>>>>> "filestore_update_to": "1000",
>>>>> "filestore_blackhole": "false",
>>>>> "filestore_dump_file": "",
>>>>> "filestore_kill_at": "0",
>>>>> "filestore_fail_eio": "true",
>>>>> "journal_dio": "true",
>>>>> "journal_aio": "false",
>>>>> "journal_block_align": "true",
>>>>> "journal_max_write_bytes": "10485760",
>>>>> "journal_max_write_entries": "100",
>>>>> "journal_queue_max_ops": "500",
>>>>> "journal_queue_max_bytes": "104857600",
>>>>> "journal_align_min_size": "65536",
>>>>> "journal_replay_from": "0",
>>>>> "journal_zero_on_create": "false",
>>>>> "rbd_cache": "false",
>>>>> "rbd_cache_size": "33554432",
>>>>> "rbd_cache_max_dirty": "25165824",
>>>>> "rbd_cache_target_dirty": "16777216",
>>>>> "rbd_cache_max_dirty_age": "1",
>>>>> "nss_db_path": "",
>>>>> "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-24",
>>>>> "rgw_enable_apis": "s3, swift, swift_auth, admin",
>>>>> "rgw_cache_enabled": "true",
>>>>> "rgw_cache_lru_size": "10000",
>>>>> "rgw_socket_path": "",
>>>>> "rgw_dns_name": "",
>>>>> "rgw_swift_url": "",
>>>>> "rgw_swift_url_prefix": "swift",
>>>>> "rgw_swift_auth_url": "",
>>>>> "rgw_swift_auth_entry": "auth",
>>>>> "rgw_keystone_url": "",
>>>>> "rgw_keystone_admin_token": "",
>>>>> "rgw_keystone_accepted_roles": "Member, admin",
>>>>> "rgw_keystone_token_cache_size": "10000",
>>>>> "rgw_keystone_revocation_interval": "900",
>>>>> "rgw_admin_entry": "admin",
>>>>> "rgw_enforce_swift_acls": "true",
>>>>> "rgw_swift_token_expiration": "86400",
>>>>> "rgw_print_continue": "true",
>>>>> "rgw_remote_addr_param": "REMOTE_ADDR",
>>>>> "rgw_op_thread_timeout": "600",
>>>>> "rgw_op_thread_suicide_timeout": "0",
>>>>> "rgw_thread_pool_size": "100",
>>>>> "rgw_num_control_oids": "8",
>>>>> "rgw_cluster_root_pool": ".rgw.root",
>>>>> "rgw_log_nonexistent_bucket": "false",
>>>>> "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
>>>>> "rgw_log_object_name_utc": "false",
>>>>> "rgw_usage_max_shards": "32",
>>>>> "rgw_usage_max_user_shards": "1",
>>>>> "rgw_enable_ops_log": "true",
>>>>> "rgw_enable_usage_log": "true",
>>>>> "rgw_ops_log_rados": "true",
>>>>> "rgw_ops_log_socket_path": "",
>>>>> "rgw_ops_log_data_backlog": "5242880",
>>>>> "rgw_usage_log_flush_threshold": "1024",
>>>>> "rgw_usage_log_tick_interval": "30",
>>>>> "rgw_intent_log_object_name": "%Y-%m-%d-%i-%n",
>>>>> "rgw_intent_log_object_name_utc": "false",
>>>>> "rgw_init_timeout": "30",
>>>>> "rgw_mime_types_file": "\/etc\/mime.types",
>>>>> "rgw_gc_max_objs": "32",
>>>>> "rgw_gc_obj_min_wait": "7200",
>>>>> "rgw_gc_processor_max_time": "3600",
>>>>> "rgw_gc_processor_period": "3600",
>>>>> "rgw_s3_success_create_obj_status": "0",
>>>>> "rgw_resolve_cname": "false",
>>>>> "rgw_obj_stripe_size": "4194304",
>>>>> "rgw_extended_http_attrs": "",
>>>>> "mutex_perf_counter": "false",
>>>>> "internal_safe_to_start_threads": "true"}
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Mark Nelson [mailto:mark.nelson@inktank.com]
>>>>> Sent: Thursday, 20 December 2012 11:28 PM
>>>>> To: Matthew Anderson
>>>>> Cc: ceph-devel@vger.kernel.org
>>>>> Subject: Re: OSD's slow down to a crawl
>>>>>
>>>>> Hi Matt,
>>>>>
>>>>> Yeah, your log is looking pretty sparse! I imagine Sam will have a lot of questions for you. Out of curiosity have you looked at any io/cpu/memory stats when things slow down? sysprof may come in handy if it looks like it's CPU related. Also, check out the OSD admin socket.
>>>>> Sebastien Han made a nice little blog entry describing it:
>>>>>
>>>>> http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
>>>>>
>>>>> You can get a lot of information about the current state of things.
>>>>>
>>>>> Mark
>>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>> in the body of a message to majordomo@vger.kernel.org More
>>>> majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>
--
Mark Nelson
Performance Engineer
Inktank
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2013-01-09 14:10 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-20 15:16 OSD's slow down to a crawl Matthew Anderson
2012-12-20 15:28 ` Mark Nelson
2012-12-20 15:51 ` Matthew Anderson
2012-12-20 15:59 ` Mark Nelson
2012-12-20 16:08 ` Matthew Anderson
2012-12-20 16:22 ` Mark Nelson
2012-12-20 16:34 ` Matthew Anderson
2012-12-20 16:30 ` Sage Weil
[not found] ` <38A500831D3DE24B90BD200D6C8701351BB15433@Exchange2010-2.corit.local>
2012-12-20 17:13 ` Sage Weil
2012-12-21 4:51 ` Matthew Anderson
2012-12-21 16:02 ` Sage Weil
2013-01-09 8:52 ` Matthew Anderson
2013-01-09 14:09 ` Mark Nelson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.