Re: v0.56 released

[norbi <norbi@rocknob.de>]

Hi List,

after upgrading from 0.55.1 to 0.56 some MONs are crashing during the 
upgrade.

I have 3 MONs with 0.55.1, mon.a, mon.b. and mon.c

So now i am upgrading mon.a to 0.56, i restarted mon.a and see that 
mon.c is crashed... so i restarted mon.c and see, now mon.b is crashed, 
after restart all mons are running ?

The Log from mon.b


     -7> 2013-01-03 09:09:02.011229 7fc4d1d00700 -1 mon/PaxosService.cc: 
In function 'void PaxosService::propose_pending()' thread 7fc4d1d00700 
time 2013-01-03 09:09:01.900100
mon/PaxosService.cc: 110: FAILED assert(have_pending)

  ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
  1: /usr/local/bin/ceph-mon() [0x4a6e94]
  2: (MDSMonitor::tick()+0x1a45) [0x4e1245]
  3: (MDSMonitor::on_active()+0x1f) [0x4d67ef]
  4: (PaxosService::_active()+0x245) [0x4a7a95]
  5: (Context::complete(int)+0xa) [0x48bbda]
  6: (finish_contexts(CephContext*, std::list<Context*, 
std::allocator<Context*> >&, int)+0x122) [0x496d72]
  7: (Monitor::recovered_leader(int)+0x378) [0x478ed8]
  8: (Paxos::handle_last(MMonPaxos*)+0xb19) [0x4a3919]
  9: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4a40fb]
  10: (Monitor::_ms_dispatch(Message*)+0x1298) [0x48ae78]
  11: (Monitor::ms_dispatch(Message*)+0x32) [0x49a932]
  12: (DispatchQueue::entry()+0x2d9) [0x620c19]
  13: (DispatchQueue::DispatchThread::entry()+0xd) [0x5c3a8d]
  14: (()+0x7851) [0x7fc4d65e6851]
  15: (clone()+0x6d) [0x7fc4d4df011d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is 
needed to interpret this.

     -6> 2013-01-03 09:09:02.044710 7fc4cf7e9700  1 -- 
46.252.23.110:6789/0 >> :/0 pipe(0x477e540 sd=26 :6789 pgs=0 cs=0 
l=0).accept sd=26
     -5> 2013-01-03 09:09:02.219117 7fc4cf4e6700  1 -- 
46.252.23.110:6789/0 >> :/0 pipe(0x4778480 sd=28 :6789 pgs=0 cs=0 
l=0).accept sd=28
     -4> 2013-01-03 09:09:02.462884 7fc4cf3e5700  1 -- 
46.252.23.110:6789/0 >> :/0 pipe(0x4718240 sd=29 :6789 pgs=0 cs=0 
l=0).accept sd=29
     -3> 2013-01-03 09:09:02.848348 7fc4cfcee700  1 -- 
46.252.23.110:6789/0 >> :/0 pipe(0x4718000 sd=30 :6789 pgs=0 cs=0 
l=0).accept sd=30
     -2> 2013-01-03 09:09:02.924980 7fc4ceddf700  2 -- 
46.252.23.110:6789/0 >> 80.67.16.129:6800/31582 pipe(0x471a640 sd=17 
:6789 pgs=22 cs=1 l=1).reader couldn't read tag, Success
     -1> 2013-01-03 09:09:02.925020 7fc4ceddf700  2 -- 
46.252.23.110:6789/0 >> 80.67.16.129:6800/31582 pipe(0x471a640 sd=17 
:6789 pgs=22 cs=1 l=1).fault 0: Success
--- logging levels ---
    0/ 5 none
    0/ 1 lockdep
    0/ 1 context
    1/ 1 crush
    1/ 5 mds
    1/ 5 mds_balancer
    1/ 5 mds_locker
    1/ 5 mds_log
    1/ 5 mds_log_expire
    1/ 5 mds_migrator
    0/ 1 buffer
    0/ 1 timer
    0/ 1 filer
    0/ 1 striper
    0/ 1 objecter
    0/ 5 rados
    0/ 5 rbd
    0/ 5 journaler
    0/ 5 objectcacher
    0/ 5 client
    0/ 5 osd
    0/ 5 optracker
    0/ 5 objclass
    1/ 3 filestore
    1/ 3 journal
    0/ 5 ms
    1/ 5 mon
    0/10 monc
    0/ 5 paxos
    0/ 5 tp
    1/ 5 auth
    1/ 5 crypto
    1/ 1 finisher
    1/ 5 heartbeatmap
    1/ 5 perfcounter
    1/ 5 rgw
    1/ 5 hadoop
    1/ 5 javaclient
    1/ 5 asok
    1/ 1 throttle
   -2/-2 (syslog threshold)
   -1/-1 (stderr threshold)
   max_recent    100000
   max_new         1000
   log_file /var/log/ceph/mon.b.log
--- end dump of recent events ---
2013-01-03 09:09:03.039368 7fc4d1d00700 -1 *** Caught signal (Aborted) **
  in thread 7fc4d1d00700

  ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
  1: /usr/local/bin/ceph-mon() [0x537729]
  2: (()+0xf500) [0x7fc4d65ee500]
  3: (gsignal()+0x35) [0x7fc4d4d3a8a5]
  4: (abort()+0x175) [0x7fc4d4d3c085]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fc4d55f3a5d]
  6: (()+0xbcbe6) [0x7fc4d55f1be6]
  7: (()+0xbcc13) [0x7fc4d55f1c13]
  8: (()+0xbcd0e) [0x7fc4d55f1d0e]
  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x7c9) [0x5cfe39]
  10: /usr/local/bin/ceph-mon() [0x4a6e94]
  11: (MDSMonitor::tick()+0x1a45) [0x4e1245]
  12: (MDSMonitor::on_active()+0x1f) [0x4d67ef]
  13: (PaxosService::_active()+0x245) [0x4a7a95]
  14: (Context::complete(int)+0xa) [0x48bbda]
  15: (finish_contexts(CephContext*, std::list<Context*, 
std::allocator<Context*> >&, int)+0x122) [0x496d72]
  16: (Monitor::recovered_leader(int)+0x378) [0x478ed8]
  17: (Paxos::handle_last(MMonPaxos*)+0xb19) [0x4a3919]
  18: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4a40fb]
  19: (Monitor::_ms_dispatch(Message*)+0x1298) [0x48ae78]
  20: (Monitor::ms_dispatch(Message*)+0x32) [0x49a932]
  21: (DispatchQueue::entry()+0x2d9) [0x620c19]
  22: (DispatchQueue::DispatchThread::entry()+0xd) [0x5c3a8d]
  23: (()+0x7851) [0x7fc4d65e6851]
  24: (clone()+0x6d) [0x7fc4d4df011d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is 
needed to interpret this.

--- begin dump of recent events ---
     -1> 2013-01-03 09:09:03.039368 7fc4d1d00700 -1 *** Caught signal 
(Aborted) **
  in thread 7fc4d1d00700

  ceph version 0.55.1 (8e25c8d984f9258644389a18997ec6bdef8e056b)
  1: /usr/local/bin/ceph-mon() [0x537729]
  2: (()+0xf500) [0x7fc4d65ee500]
  3: (gsignal()+0x35) [0x7fc4d4d3a8a5]
  4: (abort()+0x175) [0x7fc4d4d3c085]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7fc4d55f3a5d]
  6: (()+0xbcbe6) [0x7fc4d55f1be6]
  7: (()+0xbcc13) [0x7fc4d55f1c13]
  8: (()+0xbcd0e) [0x7fc4d55f1d0e]
  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x7c9) [0x5cfe39]
  10: /usr/local/bin/ceph-mon() [0x4a6e94]
  11: (MDSMonitor::tick()+0x1a45) [0x4e1245]
  12: (MDSMonitor::on_active()+0x1f) [0x4d67ef]
  13: (PaxosService::_active()+0x245) [0x4a7a95]
  14: (Context::complete(int)+0xa) [0x48bbda]
  15: (finish_contexts(CephContext*, std::list<Context*, 
std::allocator<Context*> >&, int)+0x122) [0x496d72]
  16: (Monitor::recovered_leader(int)+0x378) [0x478ed8]
  17: (Paxos::handle_last(MMonPaxos*)+0xb19) [0x4a3919]
  18: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4a40fb]
  19: (Monitor::_ms_dispatch(Message*)+0x1298) [0x48ae78]
  20: (Monitor::ms_dispatch(Message*)+0x32) [0x49a932]
  21: (DispatchQueue::entry()+0x2d9) [0x620c19]
  22: (DispatchQueue::DispatchThread::entry()+0xd) [0x5c3a8d]
  23: (()+0x7851) [0x7fc4d65e6851]
  24: (clone()+0x6d) [0x7fc4d4df011d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is 
needed to interpret this.

--- logging levels ---
    0/ 5 none
    0/ 1 lockdep
    0/ 1 context
    1/ 1 crush
    1/ 5 mds
    1/ 5 mds_balancer
    1/ 5 mds_locker
    1/ 5 mds_log
    1/ 5 mds_log_expire
    1/ 5 mds_migrator
    0/ 1 buffer
    0/ 1 timer
    0/ 1 filer
    0/ 1 striper
    0/ 1 objecter
    0/ 5 rados
    0/ 5 rbd
    0/ 5 journaler
    0/ 5 objectcacher
    0/ 5 client
    0/ 5 osd
    0/ 5 optracker
    0/ 5 objclass
    1/ 3 filestore
    1/ 3 journal
    0/ 5 ms
    1/ 5 mon
    0/10 monc
    0/ 5 paxos
    0/ 5 tp
    1/ 5 auth
    1/ 5 crypto
    1/ 1 finisher
    1/ 5 heartbeatmap
    1/ 5 perfcounter
    1/ 5 rgw
    1/ 5 hadoop
    1/ 5 javaclient
    1/ 5 asok
    1/ 1 throttle
   -2/-2 (syslog threshold)
   -1/-1 (stderr threshold)
   max_recent    100000
   max_new         1000
   log_file /var/log/ceph/mon.b.log
--- end dump of recent events ---


Norbert

On 01.01.2013 07:02, Sage Weil wrote:
> We're bringing in the new year with a new release, v0.56, which will form
> the basis of the next stable series "bobtail." There is little in the way
> of new functionality since v0.55, as we've been focusing primarily on
> stability, performance, and upgradability from the previous argonaut
> stable series (v0.48.x). If you are a current argonaut user, you can
> either upgrade now, or watch the Inktank blog for the bobtail announcement
> after some additional testing has been completed. If you are a v0.55 or
> v0.55.1 user, we recommend upgrading now.
>
> Notable changes since v0.55 include:
>
>   * librbd: fixes for read-only pools for image cloning
>   * osd: fix for mixing argonaut and post-v0.54 OSDs
>   * osd: some recovery tuning
>   * osd: fix for several scrub, recovery, and watch/notify races/bugs
>   * osd: fix pool_stat_t backwawrd compatibility with pre-v0.41 clients
>   * osd: experimental split support
>   * mkcephfs: misc fixes for fs initialization, mounting
>   * radosgw: usage and op logs off by default
>   * radosgw: keystone authentication off by default
>   * upstart: only enabled with 'upstart' file exists in daemon data
>     directory
>   * mount.fuse.ceph: allow mounting of ceph-fuse via /etc/fstab
>   * config: always complain about config parsing errors
>   * mon: fixed memory leaks, misc bugs
>   * mds: many misc fixes
>
> Notable changes since v0.48.2 ("argonaut"):
>
>   * auth: authentication is now on by default; see release notes!
>   * osd: improved threading, small io performance
>   * osd: deep scrubbing (verify object data)
>   * osd: chunky scrubs (more efficient)
>   * osd: improved performance during recovery
>   * librbd: cloning support
>   * librbd: fine-grained striping support
>   * librbd: better caching
>   * radosgw: improved Swift and S3 API coverage (POST, multi-object delete,
>     striping)
>   * radosgw: OpenStack Keystone integration
>   * radosgw: efficient usage stats aggregation (for billing)
>   * crush: improvements in distribution (still off by default; see CRUSH
>     tunables)
>   * ceph-fuse, mds: general stability improvements
>   * release RPMs for OpenSUSE, SLES, Fedora, RHEL, CentOS
>   * tons and bug fixes and small improvements across the board
>
> If you are upgrading from v0.55, there are no special upgrade
> instructions. If you are upgrading from an older version, please read the
> release notes. Authentication is now enabled by default, and if you do not
> adjust your ceph.conf accordingly before upgrading the system will not
> come up by itself.
>
> You can get this release from the usual locations:
>
>   * Git at git://github.com/ceph/ceph.git
>   * Tarball at http://ceph.com/download/ceph-0.56.tar.gz
>   * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>   * For RPMs, see http://ceph.com/docs/master/install/rpm
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>