From: Joao Eduardo Luis <joao.luis@inktank.com>
To: norbi <norbi@rocknob.de>
Cc: ceph-devel@vger.kernel.org
Subject: Re: v0.56 released
Date: Thu, 03 Jan 2013 10:44:41 +0000 [thread overview]
Message-ID: <50E56119.9070608@inktank.com> (raw)
In-Reply-To: <50E54110.8090203@rocknob.de>
On 01/03/2013 08:28 AM, norbi wrote:
> Hi List,
>
> after upgrading from 0.55.1 to 0.56 some MONs are crashing during the
> upgrade.
>
> I have 3 MONs with 0.55.1, mon.a, mon.b. and mon.c
>
> So now i am upgrading mon.a to 0.56, i restarted mon.a and see that
> mon.c is crashed... so i restarted mon.c and see, now mon.b is crashed,
> after restart all mons are running ?
>
> The Log from mon.b
>
Hello Norbert,
You hit a bug [1] still present on 0.55.1 but fixed on 0.56.
[1] - http://tracker.newdream.net/issues/3495
-Joao
>
> -7> 2013-01-03 09:09:02.011229 7fc4d1d00700 -1 mon/PaxosService.cc:
> In function 'void PaxosService::propose_pending()' thread 7fc4d1d00700
> time 2013-01-03 09:09:01.900100
> mon/PaxosService.cc: 110: FAILED assert(have_pending)
>
> ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
> 1: /usr/local/bin/ceph-mon() [0x4a6e94]
> 2: (MDSMonitor::tick()+0x1a45) [0x4e1245]
> 3: (MDSMonitor::on_active()+0x1f) [0x4d67ef]
> 4: (PaxosService::_active()+0x245) [0x4a7a95]
> 5: (Context::complete(int)+0xa) [0x48bbda]
> 6: (finish_contexts(CephContext*, std::list<Context*,
> std::allocator<Context*> >&, int)+0x122) [0x496d72]
> 7: (Monitor::recovered_leader(int)+0x378) [0x478ed8]
> 8: (Paxos::handle_last(MMonPaxos*)+0xb19) [0x4a3919]
> 9: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4a40fb]
> 10: (Monitor::_ms_dispatch(Message*)+0x1298) [0x48ae78]
> 11: (Monitor::ms_dispatch(Message*)+0x32) [0x49a932]
> 12: (DispatchQueue::entry()+0x2d9) [0x620c19]
> 13: (DispatchQueue::DispatchThread::entry()+0xd) [0x5c3a8d]
> 14: (()+0x7851) [0x7fc4d65e6851]
> 15: (clone()+0x6d) [0x7fc4d4df011d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> -6> 2013-01-03 09:09:02.044710 7fc4cf7e9700 1 --
> 46.252.23.110:6789/0 >> :/0 pipe(0x477e540 sd=26 :6789 pgs=0 cs=0
> l=0).accept sd=26
> -5> 2013-01-03 09:09:02.219117 7fc4cf4e6700 1 --
> 46.252.23.110:6789/0 >> :/0 pipe(0x4778480 sd=28 :6789 pgs=0 cs=0
> l=0).accept sd=28
> -4> 2013-01-03 09:09:02.462884 7fc4cf3e5700 1 --
> 46.252.23.110:6789/0 >> :/0 pipe(0x4718240 sd=29 :6789 pgs=0 cs=0
> l=0).accept sd=29
> -3> 2013-01-03 09:09:02.848348 7fc4cfcee700 1 --
> 46.252.23.110:6789/0 >> :/0 pipe(0x4718000 sd=30 :6789 pgs=0 cs=0
> l=0).accept sd=30
> -2> 2013-01-03 09:09:02.924980 7fc4ceddf700 2 --
> 46.252.23.110:6789/0 >> 80.67.16.129:6800/31582 pipe(0x471a640 sd=17
> :6789 pgs=22 cs=1 l=1).reader couldn't read tag, Success
> -1> 2013-01-03 09:09:02.925020 7fc4ceddf700 2 --
> 46.252.23.110:6789/0 >> 80.67.16.129:6800/31582 pipe(0x471a640 sd=17
> :6789 pgs=22 cs=1 l=1).fault 0: Success
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 0/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/ 5 hadoop
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 100000
> max_new 1000
> log_file /var/log/ceph/mon.b.log
> --- end dump of recent events ---
> 2013-01-03 09:09:03.039368 7fc4d1d00700 -1 *** Caught signal (Aborted) **
> in thread 7fc4d1d00700
>
> ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
> 1: /usr/local/bin/ceph-mon() [0x537729]
> 2: (()+0xf500) [0x7fc4d65ee500]
> 3: (gsignal()+0x35) [0x7fc4d4d3a8a5]
> 4: (abort()+0x175) [0x7fc4d4d3c085]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fc4d55f3a5d]
> 6: (()+0xbcbe6) [0x7fc4d55f1be6]
> 7: (()+0xbcc13) [0x7fc4d55f1c13]
> 8: (()+0xbcd0e) [0x7fc4d55f1d0e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x7c9) [0x5cfe39]
> 10: /usr/local/bin/ceph-mon() [0x4a6e94]
> 11: (MDSMonitor::tick()+0x1a45) [0x4e1245]
> 12: (MDSMonitor::on_active()+0x1f) [0x4d67ef]
> 13: (PaxosService::_active()+0x245) [0x4a7a95]
> 14: (Context::complete(int)+0xa) [0x48bbda]
> 15: (finish_contexts(CephContext*, std::list<Context*,
> std::allocator<Context*> >&, int)+0x122) [0x496d72]
> 16: (Monitor::recovered_leader(int)+0x378) [0x478ed8]
> 17: (Paxos::handle_last(MMonPaxos*)+0xb19) [0x4a3919]
> 18: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4a40fb]
> 19: (Monitor::_ms_dispatch(Message*)+0x1298) [0x48ae78]
> 20: (Monitor::ms_dispatch(Message*)+0x32) [0x49a932]
> 21: (DispatchQueue::entry()+0x2d9) [0x620c19]
> 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x5c3a8d]
> 23: (()+0x7851) [0x7fc4d65e6851]
> 24: (clone()+0x6d) [0x7fc4d4df011d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- begin dump of recent events ---
> -1> 2013-01-03 09:09:03.039368 7fc4d1d00700 -1 *** Caught signal
> (Aborted) **
> in thread 7fc4d1d00700
>
> ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
> 1: /usr/local/bin/ceph-mon() [0x537729]
> 2: (()+0xf500) [0x7fc4d65ee500]
> 3: (gsignal()+0x35) [0x7fc4d4d3a8a5]
> 4: (abort()+0x175) [0x7fc4d4d3c085]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fc4d55f3a5d]
> 6: (()+0xbcbe6) [0x7fc4d55f1be6]
> 7: (()+0xbcc13) [0x7fc4d55f1c13]
> 8: (()+0xbcd0e) [0x7fc4d55f1d0e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x7c9) [0x5cfe39]
> 10: /usr/local/bin/ceph-mon() [0x4a6e94]
> 11: (MDSMonitor::tick()+0x1a45) [0x4e1245]
> 12: (MDSMonitor::on_active()+0x1f) [0x4d67ef]
> 13: (PaxosService::_active()+0x245) [0x4a7a95]
> 14: (Context::complete(int)+0xa) [0x48bbda]
> 15: (finish_contexts(CephContext*, std::list<Context*,
> std::allocator<Context*> >&, int)+0x122) [0x496d72]
> 16: (Monitor::recovered_leader(int)+0x378) [0x478ed8]
> 17: (Paxos::handle_last(MMonPaxos*)+0xb19) [0x4a3919]
> 18: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4a40fb]
> 19: (Monitor::_ms_dispatch(Message*)+0x1298) [0x48ae78]
> 20: (Monitor::ms_dispatch(Message*)+0x32) [0x49a932]
> 21: (DispatchQueue::entry()+0x2d9) [0x620c19]
> 22: (DispatchQueue::DispatchThread::entry()+0xd) [0x5c3a8d]
> 23: (()+0x7851) [0x7fc4d65e6851]
> 24: (clone()+0x6d) [0x7fc4d4df011d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
prev parent reply other threads:[~2013-01-03 10:45 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-01 6:02 v0.56 released Sage Weil
2013-01-02 2:24 ` Dennis Jacobfeuerborn
2013-01-02 2:28 ` Mark Nelson
2013-01-02 9:45 ` Sébastien Han
2013-01-02 17:35 ` Sage Weil
2013-01-02 18:03 ` Sébastien Han
2013-01-02 18:11 ` Sage Weil
2013-01-02 18:14 ` Sébastien Han
2013-01-02 20:27 ` Dennis Jacobfeuerborn
2013-01-02 20:46 ` Sage Weil
2013-01-03 4:00 ` Dennis Jacobfeuerborn
2013-01-03 5:49 ` Dan Mick
2013-01-03 8:28 ` norbi
2013-01-03 10:44 ` Joao Eduardo Luis [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50E56119.9070608@inktank.com \
--to=joao.luis@inktank.com \
--cc=ceph-devel@vger.kernel.org \
--cc=norbi@rocknob.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.