From: "Dave Humphreys (Datatone)" <dave@datatone.co.uk>
To: ceph-devel@vger.kernel.org
Subject: Can't start ceph mon
Date: Mon, 19 Nov 2012 16:08:56 -0500 [thread overview]
Message-ID: <3649717.axTEZpMgMX@vault01> (raw)
I have a problem in which I can't start my ceph monitor. The log is shown below.
The log shows version 0.54. I was running 0.52 when the problem arose, and I moved to the latest in case the newer version fixed the problem.
The original failure happened a week or so ago, and could have been as a result of running out of disk space when the ceph monitor log became huge.
What should I do to recover the situation?
David
2012-11-19 20:38:51.598468 7fc13fdc6780 0 ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150), process ceph-mon, pid 21012
2012-11-19 20:38:51.598482 7fc13fdc6780 1 store(/ceph/mon.vault01) mount
2012-11-19 20:38:51.598527 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 21
2012-11-19 20:38:51.598542 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl magic = 21 bytes
2012-11-19 20:38:51.598562 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 75
2012-11-19 20:38:51.598567 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl feature_set = 75 bytes
2012-11-19 20:38:51.598582 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 205
2012-11-19 20:38:51.598586 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl monmap/latest = 205 bytes
2012-11-19 20:38:51.598809 7fc13fdc6780 1 -- 10.0.1.1:6789/0 learned my addr 10.0.1.1:6789/0
2012-11-19 20:38:51.598818 7fc13fdc6780 1 accepter.accepter.bind my_inst.addr is 10.0.1.1:6789/0 need_addr=0
2012-11-19 20:38:51.599498 7fc13fdc6780 1 -- 10.0.1.1:6789/0 messenger.start
2012-11-19 20:38:51.599508 7fc13fdc6780 1 accepter.accepter.start
2012-11-19 20:38:51.599610 7fc13fdc6780 1 mon.vault01@-1(probing) e1 init fsid 4d7d8d20-338c-4bdc-9918-9bcf04f9a832
2012-11-19 20:38:51.599674 7fc13cdbe700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213c6c0 sd=14 :6789 pgs=0 cs=0 l=0).accept sd=14
2012-11-19 20:38:51.599678 7fc141eff700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213c240 sd=9 :6789 pgs=0 cs=0 l=0).accept sd=9
2012-11-19 20:38:51.599718 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 37
2012-11-19 20:38:51.599723 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl cluster_uuid = 37 bytes
2012-11-19 20:38:51.599718 7fc13ccbd700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213c480 sd=19 :6789 pgs=0 cs=0 l=0).accept sd=19
2012-11-19 20:38:51.599729 7fc13fdc6780 10 mon.vault01@-1(probing) e1 check_fsid cluster_uuid contains '4d7d8d20-338c-4bdc-9918-9bcf04f9a832'
2012-11-19 20:38:51.599739 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 75
2012-11-19 20:38:51.599745 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl feature_set = 75 bytes
2012-11-19 20:38:51.599751 7fc13fdc6780 10 mon.vault01@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18)}
2012-11-19 20:38:51.599759 7fc13fdc6780 15 store(/ceph/mon.vault01) exists_bl joined
2012-11-19 20:38:51.599769 7fc13fdc6780 10 mon.vault01@-1(probing) e1 has_ever_joined = 1
2012-11-19 20:38:51.599794 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int pgmap/last_committed = 133333
2012-11-19 20:38:51.599801 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int pgmap/first_committed = 132833
2012-11-19 20:38:51.599810 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 239840
2012-11-19 20:38:51.599928 7fc13cbbc700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213cd80 sd=20 :6789 pgs=0 cs=0 l=0).accept sd=20
2012-11-19 20:38:51.599950 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl pgmap/latest = 239840 bytes
--- begin dump of recent events ---2012-11-19 20:38:51.600509 7fc13fdc6780 -1
*** Caught signal (Aborted) **
in thread 7fc13fdc6780
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)
1: ceph-mon() [0x53adf8]
2: (()+0xfe90) [0x7fc141830e90]
3: (gsignal()+0x3e) [0x7fc140016dae]
4: (abort()+0x17b) [0x7fc14001825b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc141af300d]
6: (()+0xb31b6) [0x7fc141af11b6]
7: (()+0xb31e3) [0x7fc141af11e3]
8: (()+0xb32de) [0x7fc141af12de]
9: ceph-mon() [0x5ecb9f]
10: (Paxos::get_stashed(ceph::buffer::list&)+0x1ed) [0x49e28d]
11: (Paxos::init()+0x109) [0x49e609]
12: (Monitor::init()+0x36a) [0x485a4a]
13: (main()+0x1289) [0x46d909]
14: (__libc_start_main()+0xed) [0x7fc14000364d]
15: ceph-mon() [0x46fa09]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-55> 2012-11-19 20:38:51.596694 7fc13fdc6780 5 asok(0x213d000) register_command perfcounters_dump hook 0x2131050
-55> 2012-11-19 20:38:51.596720 7fc13fdc6780 5 asok(0x213d000) register_command 1 hook 0x2131050
-54> 2012-11-19 20:38:51.596725 7fc13fdc6780 5 asok(0x213d000) register_command perf dump hook 0x2131050
-53> 2012-11-19 20:38:51.596735 7fc13fdc6780 5 asok(0x213d000) register_command perfcounters_schema hook 0x2131050
-52> 2012-11-19 20:38:51.596740 7fc13fdc6780 5 asok(0x213d000) register_command 2 hook 0x2131050
-51> 2012-11-19 20:38:51.596745 7fc13fdc6780 5 asok(0x213d000) register_command perf schema hook 0x2131050
-50> 2012-11-19 20:38:51.596752 7fc13fdc6780 5 asok(0x213d000) register_command config show hook 0x2131050
-49> 2012-11-19 20:38:51.596756 7fc13fdc6780 5 asok(0x213d000) register_command config set hook 0x2131050
-48> 2012-11-19 20:38:51.596761 7fc13fdc6780 5 asok(0x213d000) register_command log flush hook 0x2131050
-47> 2012-11-19 20:38:51.596765 7fc13fdc6780 5 asok(0x213d000) register_command log dump hook 0x2131050
-46> 2012-11-19 20:38:51.596770 7fc13fdc6780 5 asok(0x213d000) register_command log reopen hook 0x2131050
-45> 2012-11-19 20:38:51.598468 7fc13fdc6780 0 ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150), process ceph-mon, pid 21012
-44> 2012-11-19 20:38:51.598482 7fc13fdc6780 1 store(/ceph/mon.vault01) mount
-43> 2012-11-19 20:38:51.598527 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 21
-42> 2012-11-19 20:38:51.598542 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl magic = 21 bytes
-41> 2012-11-19 20:38:51.598562 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 75
-40> 2012-11-19 20:38:51.598567 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl feature_set = 75 bytes
-39> 2012-11-19 20:38:51.598582 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 205
-38> 2012-11-19 20:38:51.598586 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl monmap/latest = 205 bytes
-37> 2012-11-19 20:38:51.598809 7fc13fdc6780 1 -- 10.0.1.1:6789/0 learned my addr 10.0.1.1:6789/0
-36> 2012-11-19 20:38:51.598818 7fc13fdc6780 1 accepter.accepter.bind my_inst.addr is 10.0.1.1:6789/0 need_addr=0
-35> 2012-11-19 20:38:51.599219 7fc13fdc6780 1 finished global_init_daemonize
-34> 2012-11-19 20:38:51.599350 7fc13fdc6780 5 asok(0x213d000) init /var/run/ceph/ceph-mon.vault01.asok
-33> 2012-11-19 20:38:51.599371 7fc13fdc6780 5 asok(0x213d000) bind_and_listen /var/run/ceph/ceph-mon.vault01.asok
-32> 2012-11-19 20:38:51.599438 7fc13fdc6780 5 asok(0x213d000) register_command 0 hook 0x2130030
-31> 2012-11-19 20:38:51.599444 7fc13fdc6780 5 asok(0x213d000) register_command version hook 0x2130030
-30> 2012-11-19 20:38:51.599451 7fc13fdc6780 5 asok(0x213d000) register_command git_version hook 0x2130030
-29> 2012-11-19 20:38:51.599456 7fc13fdc6780 5 asok(0x213d000) register_command help hook 0x2131040
-28> 2012-11-19 20:38:51.599472 7fc13edc2700 5 asok(0x213d000) entry start
-27> 2012-11-19 20:38:51.599498 7fc13fdc6780 1 -- 10.0.1.1:6789/0 messenger.start
-26> 2012-11-19 20:38:51.599508 7fc13fdc6780 1 accepter.accepter.start
-25> 2012-11-19 20:38:51.599610 7fc13fdc6780 1 mon.vault01@-1(probing) e1 init fsid 4d7d8d20-338c-4bdc-9918-9bcf04f9a832
-24> 2012-11-19 20:38:51.599674 7fc13cdbe700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213c6c0 sd=14 :6789 pgs=0 cs=0 l=0).accept sd=14
-23> 2012-11-19 20:38:51.599678 7fc141eff700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213c240 sd=9 :6789 pgs=0 cs=0 l=0).accept sd=9
-22> 2012-11-19 20:38:51.599718 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 37
-21> 2012-11-19 20:38:51.599723 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl cluster_uuid = 37 bytes
-20> 2012-11-19 20:38:51.599718 7fc13ccbd700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213c480 sd=19 :6789 pgs=0 cs=0 l=0).accept sd=19
-19> 2012-11-19 20:38:51.599729 7fc13fdc6780 10 mon.vault01@-1(probing) e1 check_fsid cluster_uuid contains '4d7d8d20-338c-4bdc-9918-9bcf04f9a832'
-18> 2012-11-19 20:38:51.599739 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 75
-17> 2012-11-19 20:38:51.599745 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl feature_set = 75 bytes
-16> 2012-11-19 20:38:51.599751 7fc13fdc6780 10 mon.vault01@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18)}
-15> 2012-11-19 20:38:51.599759 7fc13fdc6780 15 store(/ceph/mon.vault01) exists_bl joined
-14> 2012-11-19 20:38:51.599769 7fc13fdc6780 10 mon.vault01@-1(probing) e1 has_ever_joined = 1
-13> 2012-11-19 20:38:51.599794 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int pgmap/last_committed = 133333
-12> 2012-11-19 20:38:51.599795 7fc13cdbe700 5 throttle(mon_daemon_bytes 0x7fffae473da0) get 56 (0 -> 56)
-11> 2012-11-19 20:38:51.599801 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int pgmap/first_committed = 132833
-10> 2012-11-19 20:38:51.599805 7fc13cdbe700 5 throttle(msgr_dispatch_throttler-mon 0x2153488) get 56 (0 -> 56)
-9> 2012-11-19 20:38:51.599810 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at off 0 of 239840
-8> 2012-11-19 20:38:51.599904 7fc141eff700 5 throttle(mon_daemon_bytes 0x7fffae473da0) get 56 (56 -> 112)
-7> 2012-11-19 20:38:51.599910 7fc141eff700 5 throttle(msgr_dispatch_throttler-mon 0x2153488) get 56 (56 -> 112)
-6> 2012-11-19 20:38:51.599916 7fc13ccbd700 5 throttle(mon_daemon_bytes 0x7fffae473da0) get 62 (112 -> 174)
-5> 2012-11-19 20:38:51.599929 7fc13ccbd700 5 throttle(msgr_dispatch_throttler-mon 0x2153488) get 62 (112 -> 174)
-4> 2012-11-19 20:38:51.599928 7fc13cbbc700 1 -- 10.0.1.1:6789/0 >> :/0 pipe(0x213cd80 sd=20 :6789 pgs=0 cs=0 l=0).accept sd=20
-3> 2012-11-19 20:38:51.599950 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl pgmap/latest = 239840 bytes
-2> 2012-11-19 20:38:51.600027 7fc13cbbc700 5 throttle(mon_daemon_bytes 0x7fffae473da0) get 56 (174 -> 230)
-1> 2012-11-19 20:38:51.600033 7fc13cbbc700 5 throttle(msgr_dispatch_throttler-mon 0x2153488) get 56 (174 -> 230)
0> 2012-11-19 20:38:51.600509 7fc13fdc6780 -1 *** Caught signal (Aborted) **
in thread 7fc13fdc6780
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)
1: ceph-mon() [0x53adf8]
2: (()+0xfe90) [0x7fc141830e90]
3: (gsignal()+0x3e) [0x7fc140016dae]
4: (abort()+0x17b) [0x7fc14001825b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc141af300d]
6: (()+0xb31b6) [0x7fc141af11b6]
7: (()+0xb31e3) [0x7fc141af11e3]
8: (()+0xb32de) [0x7fc141af12de]
9: ceph-mon() [0x5ecb9f]
10: (Paxos::get_stashed(ceph::buffer::list&)+0x1ed) [0x49e28d]
11: (Paxos::init()+0x109) [0x49e609]
12: (Monitor::init()+0x36a) [0x485a4a]
13: (main()+0x1289) [0x46d909]
14: (__libc_start_main()+0xed) [0x7fc14000364d]
15: ceph-mon() [0x46fa09]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 5 lockdep
0/ 5 context
1/ 5 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 0 buffer
0/ 5 timer
0/ 5 filer
0/ 0 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 5 filestore
1/ 5 journal
1/ 1 ms
20/20 mon
0/ 5 monc
20/20 paxos
0/ 5 tp
20/20 auth
1/ 5 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 asok
1/ 5 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000000
log_file /var/log/ceph/mon.vault01.log
--- end dump of recent events ---
next reply other threads:[~2012-11-19 21:08 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-19 21:08 Dave Humphreys (Datatone) [this message]
2012-11-19 21:45 ` Can't start ceph mon Gregory Farnum
2012-11-20 2:22 ` Gregory Farnum
-- strict thread matches above, loose matches on Subject: below --
2012-11-19 21:24 Can't Start Ceph Mon Dave Humphreys (Bob)
2012-11-20 18:17 Can't start ceph mon Dave (Bob)
2012-11-20 18:26 ` Gregory Farnum
[not found] ` <50ABD056.5040103@bob-the-boat.me.uk>
[not found] ` <CAPYLRzg5EpfKqg3XNYGeWm9stNLXzpMenOdm+p6j1PafRNeNBg@mail.gmail.com>
[not found] ` <50ABD284.1060309@bob-the-boat.me.uk>
[not found] ` <CAPYLRzgGrBaL7Ny7-e7QxhTASXUFtPapGWZ2xUm7cTPhhTKbuQ@mail.gmail.com>
[not found] ` <50ABD518.3030900@bob-the-boat.me.uk>
[not found] ` <0F20B62A6EC54AD7A6845CB7BD2714B2@inktank.com>
[not found] ` <50AF32E9.2060809@bob-the-boat.me.uk>
2012-11-26 23:04 ` Gregory Farnum
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3649717.axTEZpMgMX@vault01 \
--to=dave@datatone.co.uk \
--cc=ceph-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.