MonitorStore.cc assert

MonitorStore.cc assert

[Smets, Jan (Jan)]

I was not using ceph at that time, and my clocks are drifting somehow ...

Last logs seen:

10.08.10_22:55:02.051810 7fdbdaac9710 log [WRN] : lease_expire from mon0 was sent from future time 10.08.10_22:55:03.346407, clocks not synchronized
...
10.08.11_00:16:49.492554 7fe740de6710 mds0.2  last tick was 90.001027 > 5 seconds ago, laggy_until 0.000000, setting laggy flag




mon/MonitorStore.cc: In function 'int MonitorStore::write_bl_ss(ceph::bufferlist&, const char*, const char*, bool, bool)':
mon/MonitorStore.cc:273: FAILED assert(!err)
 1: (Paxos::stash_latest(unsigned long, ceph::buffer::list&)+0x170) [0x47a780]
 2: (LogMonitor::update_from_paxos()+0x7bd) [0x4ca5cd]
 3: (Monitor::_ms_dispatch(Message*)+0x851) [0x46d791]
 4: (Monitor::ms_dispatch(Message*)+0x57) [0x4799b7]
 5: (SimpleMessenger::dispatch_entry()+0x79b) [0x453b6b]
 6: (SimpleMessenger::DispatchThread::entry()+0x1f) [0x44abbf]
 7: (Thread::_entry_func(void*)+0xa) [0x45d4da]
 8: (()+0x68ba) [0x7fdbdcd9b8ba]
 9: (clone()+0x6d) [0x7fdbdbfb601d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Executable is the one from:

ii  ceph                                0.21~rc-unstable201007281541-3ed08a33-1~bpo50+1 distributed storage and file system


At your service.
 - Jan

RE: MonitorStore.cc assert

[Smets, Jan (Jan)]

And also something similar on my other server

10.08.11_00:14:54.656802 7fbdf3c14710 log [WRN] : lease_expire from mon0 was sent from future time 10.08.11_00:14:58.132006, clocks not synchronized
10.08.11_00:14:54.656814 7fbdf3c14710 log [WRN] :
mon/MonitorStore.cc: In function 'int MonitorStore::write_bl_ss(ceph::bufferlist&, const char*, const char*, bool, bool)':
mon/MonitorStore.cc:273: FAILED assert(!err)
 1: (Paxos::store_state(MMonPaxos*)+0xe6) [0x47b5d6]
 2: (Paxos::handle_commit(MMonPaxos*)+0x3e) [0x47b8fe]
 3: (Paxos::dispatch(PaxosServiceMessage*)+0x18b) [0x47fe4b]
 4: (Monitor::_ms_dispatch(Message*)+0x82d) [0x46d76d]
 5: (Monitor::ms_dispatch(Message*)+0x57) [0x4799b7]
 6: (SimpleMessenger::dispatch_entry()+0x79b) [0x453b6b]
 7: (SimpleMessenger::DispatchThread::entry()+0x1f) [0x44abbf]
 8: (Thread::_entry_func(void*)+0xa) [0x45d4da]
 9: (()+0x68ba) [0x7fbdf5ee68ba]
 10: (clone()+0x6d) [0x7fbdf510101d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

- Jan
 

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Smets, 
> Jan (Jan)
> Sent: woensdag 11 augustus 2010 9:06
> To: ceph-devel@vger.kernel.org
> Subject: MonitorStore.cc assert
> 
> I was not using ceph at that time, and my clocks are drifting 
> somehow ...
> 
> Last logs seen:
> 
> 10.08.10_22:55:02.051810 7fdbdaac9710 log [WRN] : 
> lease_expire from mon0 was sent from future time 
> 10.08.10_22:55:03.346407, clocks not synchronized
> ...
> 10.08.11_00:16:49.492554 7fe740de6710 mds0.2  last tick was 
> 90.001027 > 5 seconds ago, laggy_until 0.000000, setting laggy flag
> 
> 
> 
> 
> mon/MonitorStore.cc: In function 'int 
> MonitorStore::write_bl_ss(ceph::bufferlist&, const char*, 
> const char*, bool, bool)':
> mon/MonitorStore.cc:273: FAILED assert(!err)
>  1: (Paxos::stash_latest(unsigned long, 
> ceph::buffer::list&)+0x170) [0x47a780]
>  2: (LogMonitor::update_from_paxos()+0x7bd) [0x4ca5cd]
>  3: (Monitor::_ms_dispatch(Message*)+0x851) [0x46d791]
>  4: (Monitor::ms_dispatch(Message*)+0x57) [0x4799b7]
>  5: (SimpleMessenger::dispatch_entry()+0x79b) [0x453b6b]
>  6: (SimpleMessenger::DispatchThread::entry()+0x1f) [0x44abbf]
>  7: (Thread::_entry_func(void*)+0xa) [0x45d4da]
>  8: (()+0x68ba) [0x7fdbdcd9b8ba]
>  9: (clone()+0x6d) [0x7fdbdbfb601d]
>  NOTE: a copy of the executable, or `objdump -rdS 
> <executable>` is needed to interpret this.
> 
> Executable is the one from:
> 
> ii  ceph                                
> 0.21~rc-unstable201007281541-3ed08a33-1~bpo50+1 distributed 
> storage and file system
> 
> 
> At your service.
>  - Jan
> --
> To unsubscribe from this list: send the line "unsubscribe 
> ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: MonitorStore.cc assert

[Sage Weil]

On Wed, 11 Aug 2010, Smets, Jan (Jan) wrote:
> I was not using ceph at that time, and my clocks are drifting somehow ...
>
> Last logs seen:
> 
> 10.08.10_22:55:02.051810 7fdbdaac9710 log [WRN] : lease_expire from mon0 was sent from future time 10.08.10_22:55:03.346407, clocks not synchronized
> ...
> 10.08.11_00:16:49.492554 7fe740de6710 mds0.2  last tick was 90.001027 > 5 seconds ago, laggy_until 0.000000, setting laggy flag
> 
> 
> 
> 
> mon/MonitorStore.cc: In function 'int MonitorStore::write_bl_ss(ceph::bufferlist&, const char*, const char*, bool, bool)':
> mon/MonitorStore.cc:273: FAILED assert(!err)
>  1: (Paxos::stash_latest(unsigned long, ceph::buffer::list&)+0x170) [0x47a780]
>  2: (LogMonitor::update_from_paxos()+0x7bd) [0x4ca5cd]
>  3: (Monitor::_ms_dispatch(Message*)+0x851) [0x46d791]
>  4: (Monitor::ms_dispatch(Message*)+0x57) [0x4799b7]
>  5: (SimpleMessenger::dispatch_entry()+0x79b) [0x453b6b]
>  6: (SimpleMessenger::DispatchThread::entry()+0x1f) [0x44abbf]
>  7: (Thread::_entry_func(void*)+0xa) [0x45d4da]
>  8: (()+0x68ba) [0x7fdbdcd9b8ba]
>  9: (clone()+0x6d) [0x7fdbdbfb601d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Usually this error means your disk filled up (ENOSPC).  Probably because 
it's filling up the logs about clock drift.  :)  We'll add something to 
throttle those.

In the meantime, you can zap the plaintext logs (rm $mon_data/log*) 
safely.

sage

Re: MonitorStore.cc assert

[Thomas Mueller]

Am Wed, 11 Aug 2010 09:06:14 +0200 schrieb Smets, Jan (Jan):

> I was not using ceph at that time, and my clocks are drifting somehow
> ...
> 
> Last logs seen:
> 
> 10.08.10_22:55:02.051810 7fdbdaac9710 log [WRN] : lease_expire from mon0
> was sent from future time 10.08.10_22:55:03.346407, clocks not
> synchronized ...

maybe not related to the problem, but also important:

do run a ntpd on every osd/mon/mds as there is a need for precise 
syncronised time. 

- Thomas