* os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
@ 2011-11-29 11:48 Martin Mailand
2011-11-29 11:56 ` Stratos Psomadakis
0 siblings, 1 reply; 9+ messages in thread
From: Martin Mailand @ 2011-11-29 11:48 UTC (permalink / raw)
To: ceph-devel
Hi
I hit this assertion a few times. I use ext4 as the osd fs, so I think
we have to replay the whole journal, maybe that triggers it.
-martin
2011-11-29 11:37:55.393296 7fab45dbc7a0 FileStore is up to date.
os/FileJournal.cc: In function 'virtual void
FileJournal::committed_thru(uint64_t)', in thread '7fab434cf700'
os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
ceph version 0.38-244-g30def38
(commit:30def38d21b217f244db74e6c469598d794fa8a1)
1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
3: (FileStore::sync_entry()+0xec7) [0x70aae7]
4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
5: (()+0x6d8c) [0x7fab45993d8c]
6: (clone()+0x6d) [0x7fab43fd004d]
ceph version 0.38-244-g30def38
(commit:30def38d21b217f244db74e6c469598d794fa8a1)
1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
3: (FileStore::sync_entry()+0xec7) [0x70aae7]
4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
5: (()+0x6d8c) [0x7fab45993d8c]
6: (clone()+0x6d) [0x7fab43fd004d]
*** Caught signal (Aborted) **
in thread 7fab434cf700
ceph version 0.38-244-g30def38
(commit:30def38d21b217f244db74e6c469598d794fa8a1)
1: /usr/bin/ceph-osd() [0x5a7ba2]
2: (()+0xfc60) [0x7fab4599cc60]
3: (gsignal()+0x35) [0x7fab43f1dd05]
4: (abort()+0x186) [0x7fab43f21ab6]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab447d46dd]
6: (()+0xb9926) [0x7fab447d2926]
7: (()+0xb9953) [0x7fab447d2953]
8: (()+0xb9a5e) [0x7fab447d2a5e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x396) [0x5cd9e6]
10: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
11: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
12: (FileStore::sync_entry()+0xec7) [0x70aae7]
13: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
14: (()+0x6d8c) [0x7fab45993d8c]
15: (clone()+0x6d) [0x7fab43fd004d]
Thread 1 (Thread 2491):
#0 0x00007fab4599cb3b in raise () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005a77f2 in reraise_fatal (signum=6) at
global/signal_handler.cc:59
#2 0x00000000005a7dea in handle_fatal_signal (signum=6) at
global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007fab43f1dd05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fab43f21ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fab447d46dd in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fab447d2926 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
---Type <return> to continue, or q <return> to quit---
#8 0x00007fab447d2953 in std::terminate() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fab447d2a5e in __cxa_throw () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00000000005cd9e6 in ceph::__ceph_assert_fail (assertion=<value
optimized out>, file=<value optimized out>, line=<value optimized out>,
func=<value optimized out>)
at common/assert.cc:70
#11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
seq=4145693) at os/FileJournal.cc:1011
#12 0x0000000000714d79 in JournalingObjectStore::commit_finish
(this=0x1401000) at os/JournalingObjectStore.cc:260
#13 0x000000000070aae7 in FileStore::sync_entry (this=0x1401000) at
os/FileStore.cc:3079
#14 0x00000000007139bd in FileStore::SyncThread::entry (this=<value
optimized out>) at os/FileStore.h:101
#15 0x00007fab45993d8c in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007fab43fd004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#17 0x0000000000000000 in ?? ()
(gdb)
(gdb) thread 1
[Switching to thread 1 (Thread 2491)]#0 0x00007fab4599cb3b in raise ()
from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) thread 11
Thread ID 11 not known.
(gdb) frame 11
#11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
seq=4145693) at os/FileJournal.cc:1011
1011 os/FileJournal.cc: No such file or directory.
in os/FileJournal.cc
(gdb) p seq
$1 = 4145693
(gdb) p last_committed_seq
$2 = 4145768
(gdb)
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-11-29 11:48 os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq) Martin Mailand
@ 2011-11-29 11:56 ` Stratos Psomadakis
2011-11-29 12:14 ` Martin Mailand
0 siblings, 1 reply; 9+ messages in thread
From: Stratos Psomadakis @ 2011-11-29 11:56 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 4854 bytes --]
On 11/29/2011 01:48 PM, Martin Mailand wrote:
> Hi
> I hit this assertion a few times. I use ext4 as the osd fs, so I think
> we have to replay the whole journal, maybe that triggers it.
>
I've hit that too with v0.38 (with OSD on ext4), but when I built ceph
from the master branch, the issue seemed to be resolved.
>
> -martin
>
> 2011-11-29 11:37:55.393296 7fab45dbc7a0 FileStore is up to date.
> os/FileJournal.cc: In function 'virtual void
> FileJournal::committed_thru(uint64_t)', in thread '7fab434cf700'
> os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
> ceph version 0.38-244-g30def38
> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
> 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
> 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
> 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
> 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
> 5: (()+0x6d8c) [0x7fab45993d8c]
> 6: (clone()+0x6d) [0x7fab43fd004d]
> ceph version 0.38-244-g30def38
> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
> 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
> 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
> 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
> 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
> 5: (()+0x6d8c) [0x7fab45993d8c]
> 6: (clone()+0x6d) [0x7fab43fd004d]
> *** Caught signal (Aborted) **
> in thread 7fab434cf700
> ceph version 0.38-244-g30def38
> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
> 1: /usr/bin/ceph-osd() [0x5a7ba2]
> 2: (()+0xfc60) [0x7fab4599cc60]
> 3: (gsignal()+0x35) [0x7fab43f1dd05]
> 4: (abort()+0x186) [0x7fab43f21ab6]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab447d46dd]
> 6: (()+0xb9926) [0x7fab447d2926]
> 7: (()+0xb9953) [0x7fab447d2953]
> 8: (()+0xb9a5e) [0x7fab447d2a5e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x396) [0x5cd9e6]
> 10: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
> 11: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
> 12: (FileStore::sync_entry()+0xec7) [0x70aae7]
> 13: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
> 14: (()+0x6d8c) [0x7fab45993d8c]
> 15: (clone()+0x6d) [0x7fab43fd004d]
>
>
> Thread 1 (Thread 2491):
> #0 0x00007fab4599cb3b in raise () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005a77f2 in reraise_fatal (signum=6) at
> global/signal_handler.cc:59
> #2 0x00000000005a7dea in handle_fatal_signal (signum=6) at
> global/signal_handler.cc:106
> #3 <signal handler called>
> #4 0x00007fab43f1dd05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #5 0x00007fab43f21ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x00007fab447d46dd in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #7 0x00007fab447d2926 in ?? () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> ---Type <return> to continue, or q <return> to quit---
> #8 0x00007fab447d2953 in std::terminate() () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #9 0x00007fab447d2a5e in __cxa_throw () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #10 0x00000000005cd9e6 in ceph::__ceph_assert_fail (assertion=<value
> optimized out>, file=<value optimized out>, line=<value optimized
> out>, func=<value optimized out>)
> at common/assert.cc:70
> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
> seq=4145693) at os/FileJournal.cc:1011
> #12 0x0000000000714d79 in JournalingObjectStore::commit_finish
> (this=0x1401000) at os/JournalingObjectStore.cc:260
> #13 0x000000000070aae7 in FileStore::sync_entry (this=0x1401000) at
> os/FileStore.cc:3079
> #14 0x00000000007139bd in FileStore::SyncThread::entry (this=<value
> optimized out>) at os/FileStore.h:101
> #15 0x00007fab45993d8c in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #16 0x00007fab43fd004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #17 0x0000000000000000 in ?? ()
> (gdb)
> (gdb) thread 1
> [Switching to thread 1 (Thread 2491)]#0 0x00007fab4599cb3b in raise
> () from /lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) thread 11
> Thread ID 11 not known.
> (gdb) frame 11
> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
> seq=4145693) at os/FileJournal.cc:1011
> 1011 os/FileJournal.cc: No such file or directory.
> in os/FileJournal.cc
> (gdb) p seq
> $1 = 4145693
> (gdb) p last_committed_seq
> $2 = 4145768
> (gdb)
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Stratos Psomadakis
<psomas@grnet.gr>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-11-29 11:56 ` Stratos Psomadakis
@ 2011-11-29 12:14 ` Martin Mailand
2011-11-29 13:56 ` Martin Mailand
0 siblings, 1 reply; 9+ messages in thread
From: Martin Mailand @ 2011-11-29 12:14 UTC (permalink / raw)
To: Stratos Psomadakis; +Cc: ceph-devel
Hi Stratos,
ok, my build was form the 23.11, I retest with master.
-martin
Am 29.11.2011 12:56, schrieb Stratos Psomadakis:
> On 11/29/2011 01:48 PM, Martin Mailand wrote:
>> Hi
>> I hit this assertion a few times. I use ext4 as the osd fs, so I think
>> we have to replay the whole journal, maybe that triggers it.
>>
> I've hit that too with v0.38 (with OSD on ext4), but when I built ceph
> from the master branch, the issue seemed to be resolved.
>
>>
>> -martin
>>
>> 2011-11-29 11:37:55.393296 7fab45dbc7a0 FileStore is up to date.
>> os/FileJournal.cc: In function 'virtual void
>> FileJournal::committed_thru(uint64_t)', in thread '7fab434cf700'
>> os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
>> ceph version 0.38-244-g30def38
>> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>> 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>> 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>> 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
>> 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>> 5: (()+0x6d8c) [0x7fab45993d8c]
>> 6: (clone()+0x6d) [0x7fab43fd004d]
>> ceph version 0.38-244-g30def38
>> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>> 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>> 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>> 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
>> 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>> 5: (()+0x6d8c) [0x7fab45993d8c]
>> 6: (clone()+0x6d) [0x7fab43fd004d]
>> *** Caught signal (Aborted) **
>> in thread 7fab434cf700
>> ceph version 0.38-244-g30def38
>> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>> 1: /usr/bin/ceph-osd() [0x5a7ba2]
>> 2: (()+0xfc60) [0x7fab4599cc60]
>> 3: (gsignal()+0x35) [0x7fab43f1dd05]
>> 4: (abort()+0x186) [0x7fab43f21ab6]
>> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab447d46dd]
>> 6: (()+0xb9926) [0x7fab447d2926]
>> 7: (()+0xb9953) [0x7fab447d2953]
>> 8: (()+0xb9a5e) [0x7fab447d2a5e]
>> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x396) [0x5cd9e6]
>> 10: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>> 11: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>> 12: (FileStore::sync_entry()+0xec7) [0x70aae7]
>> 13: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>> 14: (()+0x6d8c) [0x7fab45993d8c]
>> 15: (clone()+0x6d) [0x7fab43fd004d]
>>
>>
>> Thread 1 (Thread 2491):
>> #0 0x00007fab4599cb3b in raise () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #1 0x00000000005a77f2 in reraise_fatal (signum=6) at
>> global/signal_handler.cc:59
>> #2 0x00000000005a7dea in handle_fatal_signal (signum=6) at
>> global/signal_handler.cc:106
>> #3<signal handler called>
>> #4 0x00007fab43f1dd05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #5 0x00007fab43f21ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #6 0x00007fab447d46dd in __gnu_cxx::__verbose_terminate_handler() ()
>> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #7 0x00007fab447d2926 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> ---Type<return> to continue, or q<return> to quit---
>> #8 0x00007fab447d2953 in std::terminate() () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #9 0x00007fab447d2a5e in __cxa_throw () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #10 0x00000000005cd9e6 in ceph::__ceph_assert_fail (assertion=<value
>> optimized out>, file=<value optimized out>, line=<value optimized
>> out>, func=<value optimized out>)
>> at common/assert.cc:70
>> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
>> seq=4145693) at os/FileJournal.cc:1011
>> #12 0x0000000000714d79 in JournalingObjectStore::commit_finish
>> (this=0x1401000) at os/JournalingObjectStore.cc:260
>> #13 0x000000000070aae7 in FileStore::sync_entry (this=0x1401000) at
>> os/FileStore.cc:3079
>> #14 0x00000000007139bd in FileStore::SyncThread::entry (this=<value
>> optimized out>) at os/FileStore.h:101
>> #15 0x00007fab45993d8c in start_thread () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #16 0x00007fab43fd004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #17 0x0000000000000000 in ?? ()
>> (gdb)
>> (gdb) thread 1
>> [Switching to thread 1 (Thread 2491)]#0 0x00007fab4599cb3b in raise
>> () from /lib/x86_64-linux-gnu/libpthread.so.0
>> (gdb) thread 11
>> Thread ID 11 not known.
>> (gdb) frame 11
>> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
>> seq=4145693) at os/FileJournal.cc:1011
>> 1011 os/FileJournal.cc: No such file or directory.
>> in os/FileJournal.cc
>> (gdb) p seq
>> $1 = 4145693
>> (gdb) p last_committed_seq
>> $2 = 4145768
>> (gdb)
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-11-29 12:14 ` Martin Mailand
@ 2011-11-29 13:56 ` Martin Mailand
2011-11-29 16:07 ` Sage Weil
0 siblings, 1 reply; 9+ messages in thread
From: Martin Mailand @ 2011-11-29 13:56 UTC (permalink / raw)
To: Stratos Psomadakis; +Cc: ceph-devel
Hi,
with a build from today, I have the same prob.
os/FileJournal.cc: In function 'virtual void
FileJournal::committed_thru(uint64_t)', in thread '7fc55c85f700'
os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
ceph version 0.38-250-gc2889fe
(commit:c2889fef420611df3dd0de4064c91f6aa9f86625)
-martin
Am 29.11.2011 13:14, schrieb Martin Mailand:
> Hi Stratos,
> ok, my build was form the 23.11, I retest with master.
>
> -martin
>
> Am 29.11.2011 12:56, schrieb Stratos Psomadakis:
>> On 11/29/2011 01:48 PM, Martin Mailand wrote:
>>> Hi
>>> I hit this assertion a few times. I use ext4 as the osd fs, so I think
>>> we have to replay the whole journal, maybe that triggers it.
>>>
>> I've hit that too with v0.38 (with OSD on ext4), but when I built ceph
>> from the master branch, the issue seemed to be resolved.
>>
>>>
>>> -martin
>>>
>>> 2011-11-29 11:37:55.393296 7fab45dbc7a0 FileStore is up to date.
>>> os/FileJournal.cc: In function 'virtual void
>>> FileJournal::committed_thru(uint64_t)', in thread '7fab434cf700'
>>> os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
>>> ceph version 0.38-244-g30def38
>>> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>>> 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>>> 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>>> 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
>>> 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>>> 5: (()+0x6d8c) [0x7fab45993d8c]
>>> 6: (clone()+0x6d) [0x7fab43fd004d]
>>> ceph version 0.38-244-g30def38
>>> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>>> 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>>> 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>>> 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
>>> 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>>> 5: (()+0x6d8c) [0x7fab45993d8c]
>>> 6: (clone()+0x6d) [0x7fab43fd004d]
>>> *** Caught signal (Aborted) **
>>> in thread 7fab434cf700
>>> ceph version 0.38-244-g30def38
>>> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>>> 1: /usr/bin/ceph-osd() [0x5a7ba2]
>>> 2: (()+0xfc60) [0x7fab4599cc60]
>>> 3: (gsignal()+0x35) [0x7fab43f1dd05]
>>> 4: (abort()+0x186) [0x7fab43f21ab6]
>>> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab447d46dd]
>>> 6: (()+0xb9926) [0x7fab447d2926]
>>> 7: (()+0xb9953) [0x7fab447d2953]
>>> 8: (()+0xb9a5e) [0x7fab447d2a5e]
>>> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x396) [0x5cd9e6]
>>> 10: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>>> 11: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>>> 12: (FileStore::sync_entry()+0xec7) [0x70aae7]
>>> 13: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>>> 14: (()+0x6d8c) [0x7fab45993d8c]
>>> 15: (clone()+0x6d) [0x7fab43fd004d]
>>>
>>>
>>> Thread 1 (Thread 2491):
>>> #0 0x00007fab4599cb3b in raise () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> #1 0x00000000005a77f2 in reraise_fatal (signum=6) at
>>> global/signal_handler.cc:59
>>> #2 0x00000000005a7dea in handle_fatal_signal (signum=6) at
>>> global/signal_handler.cc:106
>>> #3<signal handler called>
>>> #4 0x00007fab43f1dd05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #5 0x00007fab43f21ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>> #6 0x00007fab447d46dd in __gnu_cxx::__verbose_terminate_handler() ()
>>> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #7 0x00007fab447d2926 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> ---Type<return> to continue, or q<return> to quit---
>>> #8 0x00007fab447d2953 in std::terminate() () from
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #9 0x00007fab447d2a5e in __cxa_throw () from
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #10 0x00000000005cd9e6 in ceph::__ceph_assert_fail (assertion=<value
>>> optimized out>, file=<value optimized out>, line=<value optimized
>>> out>, func=<value optimized out>)
>>> at common/assert.cc:70
>>> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
>>> seq=4145693) at os/FileJournal.cc:1011
>>> #12 0x0000000000714d79 in JournalingObjectStore::commit_finish
>>> (this=0x1401000) at os/JournalingObjectStore.cc:260
>>> #13 0x000000000070aae7 in FileStore::sync_entry (this=0x1401000) at
>>> os/FileStore.cc:3079
>>> #14 0x00000000007139bd in FileStore::SyncThread::entry (this=<value
>>> optimized out>) at os/FileStore.h:101
>>> #15 0x00007fab45993d8c in start_thread () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> #16 0x00007fab43fd004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>> #17 0x0000000000000000 in ?? ()
>>> (gdb)
>>> (gdb) thread 1
>>> [Switching to thread 1 (Thread 2491)]#0 0x00007fab4599cb3b in raise
>>> () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> (gdb) thread 11
>>> Thread ID 11 not known.
>>> (gdb) frame 11
>>> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
>>> seq=4145693) at os/FileJournal.cc:1011
>>> 1011 os/FileJournal.cc: No such file or directory.
>>> in os/FileJournal.cc
>>> (gdb) p seq
>>> $1 = 4145693
>>> (gdb) p last_committed_seq
>>> $2 = 4145768
>>> (gdb)
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-11-29 13:56 ` Martin Mailand
@ 2011-11-29 16:07 ` Sage Weil
2011-12-05 12:26 ` Martin Mailand
0 siblings, 1 reply; 9+ messages in thread
From: Sage Weil @ 2011-11-29 16:07 UTC (permalink / raw)
To: Martin Mailand; +Cc: Stratos Psomadakis, ceph-devel
On Tue, 29 Nov 2011, Martin Mailand wrote:
> Hi,
> with a build from today, I have the same prob.
>
> os/FileJournal.cc: In function 'virtual void
> FileJournal::committed_thru(uint64_t)', in thread '7fc55c85f700'
> os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
> ceph version 0.38-250-gc2889fe
> (commit:c2889fef420611df3dd0de4064c91f6aa9f86625)
Can you post a log of the failed ceph-osd restart with 'debug journal =
20' and 'debug filestore = 20'?
Thanks!
sage
>
> -martin
>
> Am 29.11.2011 13:14, schrieb Martin Mailand:
> > Hi Stratos,
> > ok, my build was form the 23.11, I retest with master.
> >
> > -martin
> >
> > Am 29.11.2011 12:56, schrieb Stratos Psomadakis:
> > > On 11/29/2011 01:48 PM, Martin Mailand wrote:
> > > > Hi
> > > > I hit this assertion a few times. I use ext4 as the osd fs, so I think
> > > > we have to replay the whole journal, maybe that triggers it.
> > > >
> > > I've hit that too with v0.38 (with OSD on ext4), but when I built ceph
> > > from the master branch, the issue seemed to be resolved.
> > >
> > > >
> > > > -martin
> > > >
> > > > 2011-11-29 11:37:55.393296 7fab45dbc7a0 FileStore is up to date.
> > > > os/FileJournal.cc: In function 'virtual void
> > > > FileJournal::committed_thru(uint64_t)', in thread '7fab434cf700'
> > > > os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
> > > > ceph version 0.38-244-g30def38
> > > > (commit:30def38d21b217f244db74e6c469598d794fa8a1)
> > > > 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
> > > > 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
> > > > 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
> > > > 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
> > > > 5: (()+0x6d8c) [0x7fab45993d8c]
> > > > 6: (clone()+0x6d) [0x7fab43fd004d]
> > > > ceph version 0.38-244-g30def38
> > > > (commit:30def38d21b217f244db74e6c469598d794fa8a1)
> > > > 1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
> > > > 2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
> > > > 3: (FileStore::sync_entry()+0xec7) [0x70aae7]
> > > > 4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
> > > > 5: (()+0x6d8c) [0x7fab45993d8c]
> > > > 6: (clone()+0x6d) [0x7fab43fd004d]
> > > > *** Caught signal (Aborted) **
> > > > in thread 7fab434cf700
> > > > ceph version 0.38-244-g30def38
> > > > (commit:30def38d21b217f244db74e6c469598d794fa8a1)
> > > > 1: /usr/bin/ceph-osd() [0x5a7ba2]
> > > > 2: (()+0xfc60) [0x7fab4599cc60]
> > > > 3: (gsignal()+0x35) [0x7fab43f1dd05]
> > > > 4: (abort()+0x186) [0x7fab43f21ab6]
> > > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab447d46dd]
> > > > 6: (()+0xb9926) [0x7fab447d2926]
> > > > 7: (()+0xb9953) [0x7fab447d2953]
> > > > 8: (()+0xb9a5e) [0x7fab447d2a5e]
> > > > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > > > const*)+0x396) [0x5cd9e6]
> > > > 10: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
> > > > 11: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
> > > > 12: (FileStore::sync_entry()+0xec7) [0x70aae7]
> > > > 13: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
> > > > 14: (()+0x6d8c) [0x7fab45993d8c]
> > > > 15: (clone()+0x6d) [0x7fab43fd004d]
> > > >
> > > >
> > > > Thread 1 (Thread 2491):
> > > > #0 0x00007fab4599cb3b in raise () from
> > > > /lib/x86_64-linux-gnu/libpthread.so.0
> > > > #1 0x00000000005a77f2 in reraise_fatal (signum=6) at
> > > > global/signal_handler.cc:59
> > > > #2 0x00000000005a7dea in handle_fatal_signal (signum=6) at
> > > > global/signal_handler.cc:106
> > > > #3<signal handler called>
> > > > #4 0x00007fab43f1dd05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> > > > #5 0x00007fab43f21ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> > > > #6 0x00007fab447d46dd in __gnu_cxx::__verbose_terminate_handler() ()
> > > > from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> > > > #7 0x00007fab447d2926 in ?? () from
> > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> > > > ---Type<return> to continue, or q<return> to quit---
> > > > #8 0x00007fab447d2953 in std::terminate() () from
> > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> > > > #9 0x00007fab447d2a5e in __cxa_throw () from
> > > > /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> > > > #10 0x00000000005cd9e6 in ceph::__ceph_assert_fail (assertion=<value
> > > > optimized out>, file=<value optimized out>, line=<value optimized
> > > > out>, func=<value optimized out>)
> > > > at common/assert.cc:70
> > > > #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
> > > > seq=4145693) at os/FileJournal.cc:1011
> > > > #12 0x0000000000714d79 in JournalingObjectStore::commit_finish
> > > > (this=0x1401000) at os/JournalingObjectStore.cc:260
> > > > #13 0x000000000070aae7 in FileStore::sync_entry (this=0x1401000) at
> > > > os/FileStore.cc:3079
> > > > #14 0x00000000007139bd in FileStore::SyncThread::entry (this=<value
> > > > optimized out>) at os/FileStore.h:101
> > > > #15 0x00007fab45993d8c in start_thread () from
> > > > /lib/x86_64-linux-gnu/libpthread.so.0
> > > > #16 0x00007fab43fd004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > > > #17 0x0000000000000000 in ?? ()
> > > > (gdb)
> > > > (gdb) thread 1
> > > > [Switching to thread 1 (Thread 2491)]#0 0x00007fab4599cb3b in raise
> > > > () from /lib/x86_64-linux-gnu/libpthread.so.0
> > > > (gdb) thread 11
> > > > Thread ID 11 not known.
> > > > (gdb) frame 11
> > > > #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
> > > > seq=4145693) at os/FileJournal.cc:1011
> > > > 1011 os/FileJournal.cc: No such file or directory.
> > > > in os/FileJournal.cc
> > > > (gdb) p seq
> > > > $1 = 4145693
> > > > (gdb) p last_committed_seq
> > > > $2 = 4145768
> > > > (gdb)
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> > >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-11-29 16:07 ` Sage Weil
@ 2011-12-05 12:26 ` Martin Mailand
2011-12-05 17:44 ` Sage Weil
0 siblings, 1 reply; 9+ messages in thread
From: Martin Mailand @ 2011-12-05 12:26 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]
Hi Sage,
it happened again, this time I have the log, it's attached.
(gdb) thread 1
[Switching to thread 1 (Thread 24077)]#0 0x00007f7995b83b3b in raise ()
from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) frame 11
#11 0x000000000072ee8d in FileJournal::committed_thru (this=0x1ebc000,
seq=16833973) at os/FileJournal.cc:1011
1011 os/FileJournal.cc: No such file or directory.
in os/FileJournal.cc
(gdb) p seq
$1 = 16833973
(gdb) p last_committed_seq
$2 = 16834010
(gdb)
Is this all info you need, or should I leave the osd in this state for
further debugging?
-martin
Am 29.11.2011 17:07, schrieb Sage Weil:
> On Tue, 29 Nov 2011, Martin Mailand wrote:
>> Hi,
>> with a build from today, I have the same prob.
>>
>> os/FileJournal.cc: In function 'virtual void
>> FileJournal::committed_thru(uint64_t)', in thread '7fc55c85f700'
>> os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
>> ceph version 0.38-250-gc2889fe
>> (commit:c2889fef420611df3dd0de4064c91f6aa9f86625)
>
> Can you post a log of the failed ceph-osd restart with 'debug journal =
> 20' and 'debug filestore = 20'?
>
> Thanks!
> sage
>
>
[-- Attachment #2: osd.0.log.debug.bz2 --]
[-- Type: application/x-bzip2, Size: 34629 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-12-05 12:26 ` Martin Mailand
@ 2011-12-05 17:44 ` Sage Weil
2011-12-05 18:33 ` Martin Mailand
0 siblings, 1 reply; 9+ messages in thread
From: Sage Weil @ 2011-12-05 17:44 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
dc167bac7800c75df971bded4b54e0de48f7b18f (wip-journal branch) should fix
this. Can you give it a test before I push to stable?
Thanks!
sage
On Mon, 5 Dec 2011, Martin Mailand wrote:
> Hi Sage,
> it happened again, this time I have the log, it's attached.
>
> (gdb) thread 1
> [Switching to thread 1 (Thread 24077)]#0 0x00007f7995b83b3b in raise () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) frame 11
> #11 0x000000000072ee8d in FileJournal::committed_thru (this=0x1ebc000,
> seq=16833973) at os/FileJournal.cc:1011
> 1011 os/FileJournal.cc: No such file or directory.
> in os/FileJournal.cc
> (gdb) p seq
> $1 = 16833973
> (gdb) p last_committed_seq
> $2 = 16834010
> (gdb)
>
> Is this all info you need, or should I leave the osd in this state for further
> debugging?
>
> -martin
>
> Am 29.11.2011 17:07, schrieb Sage Weil:
> > On Tue, 29 Nov 2011, Martin Mailand wrote:
> > > Hi,
> > > with a build from today, I have the same prob.
> > >
> > > os/FileJournal.cc: In function 'virtual void
> > > FileJournal::committed_thru(uint64_t)', in thread '7fc55c85f700'
> > > os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
> > > ceph version 0.38-250-gc2889fe
> > > (commit:c2889fef420611df3dd0de4064c91f6aa9f86625)
> >
> > Can you post a log of the failed ceph-osd restart with 'debug journal =
> > 20' and 'debug filestore = 20'?
> >
> > Thanks!
> > sage
> >
> >
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-12-05 17:44 ` Sage Weil
@ 2011-12-05 18:33 ` Martin Mailand
2011-12-05 19:27 ` Sage Weil
0 siblings, 1 reply; 9+ messages in thread
From: Martin Mailand @ 2011-12-05 18:33 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 1994 bytes --]
Hi Sage,
I just updated the crashed osd, and it did not work very well.
os/FileJournal.cc: 1173: FAILED assert(h->seq >= last_committed_seq)
1173 os/FileJournal.cc: No such file or directory.
in os/FileJournal.cc
(gdb) p h->seq
value has been optimized out
(gdb) p last_committed_seq
$1 = 16834095
-martin
Am 05.12.2011 18:44, schrieb Sage Weil:
> dc167bac7800c75df971bded4b54e0de48f7b18f (wip-journal branch) should fix
> this. Can you give it a test before I push to stable?
>
> Thanks!
> sage
>
>
> On Mon, 5 Dec 2011, Martin Mailand wrote:
>
>> Hi Sage,
>> it happened again, this time I have the log, it's attached.
>>
>> (gdb) thread 1
>> [Switching to thread 1 (Thread 24077)]#0 0x00007f7995b83b3b in raise () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> (gdb) frame 11
>> #11 0x000000000072ee8d in FileJournal::committed_thru (this=0x1ebc000,
>> seq=16833973) at os/FileJournal.cc:1011
>> 1011 os/FileJournal.cc: No such file or directory.
>> in os/FileJournal.cc
>> (gdb) p seq
>> $1 = 16833973
>> (gdb) p last_committed_seq
>> $2 = 16834010
>> (gdb)
>>
>> Is this all info you need, or should I leave the osd in this state for further
>> debugging?
>>
>> -martin
>>
>> Am 29.11.2011 17:07, schrieb Sage Weil:
>>> On Tue, 29 Nov 2011, Martin Mailand wrote:
>>>> Hi,
>>>> with a build from today, I have the same prob.
>>>>
>>>> os/FileJournal.cc: In function 'virtual void
>>>> FileJournal::committed_thru(uint64_t)', in thread '7fc55c85f700'
>>>> os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
>>>> ceph version 0.38-250-gc2889fe
>>>> (commit:c2889fef420611df3dd0de4064c91f6aa9f86625)
>>>
>>> Can you post a log of the failed ceph-osd restart with 'debug journal =
>>> 20' and 'debug filestore = 20'?
>>>
>>> Thanks!
>>> sage
>>>
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: osd.log.bz2 --]
[-- Type: application/x-bzip, Size: 1621 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
2011-12-05 18:33 ` Martin Mailand
@ 2011-12-05 19:27 ` Sage Weil
0 siblings, 0 replies; 9+ messages in thread
From: Sage Weil @ 2011-12-05 19:27 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
The second fix is pushed to stable branch. Thanks for testing!
(FWIW this is an old bug, so no need to rush to upgrade unless you're
actually hitting it.)
sage
On Mon, 5 Dec 2011, Martin Mailand wrote:
> Hi Sage,
> I just updated the crashed osd, and it did not work very well.
>
> os/FileJournal.cc: 1173: FAILED assert(h->seq >= last_committed_seq)
>
> 1173 os/FileJournal.cc: No such file or directory.
> in os/FileJournal.cc
> (gdb) p h->seq
> value has been optimized out
> (gdb) p last_committed_seq
> $1 = 16834095
>
>
> -martin
>
> Am 05.12.2011 18:44, schrieb Sage Weil:
> > dc167bac7800c75df971bded4b54e0de48f7b18f (wip-journal branch) should fix
> > this. Can you give it a test before I push to stable?
> >
> > Thanks!
> > sage
> >
> >
> > On Mon, 5 Dec 2011, Martin Mailand wrote:
> >
> > > Hi Sage,
> > > it happened again, this time I have the log, it's attached.
> > >
> > > (gdb) thread 1
> > > [Switching to thread 1 (Thread 24077)]#0 0x00007f7995b83b3b in raise ()
> > > from
> > > /lib/x86_64-linux-gnu/libpthread.so.0
> > > (gdb) frame 11
> > > #11 0x000000000072ee8d in FileJournal::committed_thru (this=0x1ebc000,
> > > seq=16833973) at os/FileJournal.cc:1011
> > > 1011 os/FileJournal.cc: No such file or directory.
> > > in os/FileJournal.cc
> > > (gdb) p seq
> > > $1 = 16833973
> > > (gdb) p last_committed_seq
> > > $2 = 16834010
> > > (gdb)
> > >
> > > Is this all info you need, or should I leave the osd in this state for
> > > further
> > > debugging?
> > >
> > > -martin
> > >
> > > Am 29.11.2011 17:07, schrieb Sage Weil:
> > > > On Tue, 29 Nov 2011, Martin Mailand wrote:
> > > > > Hi,
> > > > > with a build from today, I have the same prob.
> > > > >
> > > > > os/FileJournal.cc: In function 'virtual void
> > > > > FileJournal::committed_thru(uint64_t)', in thread '7fc55c85f700'
> > > > > os/FileJournal.cc: 1011: FAILED assert(seq>= last_committed_seq)
> > > > > ceph version 0.38-250-gc2889fe
> > > > > (commit:c2889fef420611df3dd0de4064c91f6aa9f86625)
> > > >
> > > > Can you post a log of the failed ceph-osd restart with 'debug journal =
> > > > 20' and 'debug filestore = 20'?
> > > >
> > > > Thanks!
> > > > sage
> > > >
> > > >
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-12-05 19:27 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-11-29 11:48 os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq) Martin Mailand
2011-11-29 11:56 ` Stratos Psomadakis
2011-11-29 12:14 ` Martin Mailand
2011-11-29 13:56 ` Martin Mailand
2011-11-29 16:07 ` Sage Weil
2011-12-05 12:26 ` Martin Mailand
2011-12-05 17:44 ` Sage Weil
2011-12-05 18:33 ` Martin Mailand
2011-12-05 19:27 ` Sage Weil
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.