Re: os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)

[Stratos Psomadakis <psomas@grnet.gr>]

[-- Attachment #1: Type: text/plain, Size: 4854 bytes --]

On 11/29/2011 01:48 PM, Martin Mailand wrote:
> Hi
> I hit this assertion a few times. I use ext4 as the osd fs, so I think
> we have to replay the whole journal, maybe that triggers it.
>
I've hit that too with v0.38 (with OSD on ext4), but when I built ceph
from the master branch, the issue seemed to be resolved.

>
> -martin
>
> 2011-11-29 11:37:55.393296 7fab45dbc7a0 FileStore is up to date.
> os/FileJournal.cc: In function 'virtual void
> FileJournal::committed_thru(uint64_t)', in thread '7fab434cf700'
> os/FileJournal.cc: 1011: FAILED assert(seq >= last_committed_seq)
>  ceph version 0.38-244-g30def38
> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>  1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>  2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>  3: (FileStore::sync_entry()+0xec7) [0x70aae7]
>  4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>  5: (()+0x6d8c) [0x7fab45993d8c]
>  6: (clone()+0x6d) [0x7fab43fd004d]
>  ceph version 0.38-244-g30def38
> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>  1: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>  2: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>  3: (FileStore::sync_entry()+0xec7) [0x70aae7]
>  4: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>  5: (()+0x6d8c) [0x7fab45993d8c]
>  6: (clone()+0x6d) [0x7fab43fd004d]
> *** Caught signal (Aborted) **
>  in thread 7fab434cf700
>  ceph version 0.38-244-g30def38
> (commit:30def38d21b217f244db74e6c469598d794fa8a1)
>  1: /usr/bin/ceph-osd() [0x5a7ba2]
>  2: (()+0xfc60) [0x7fab4599cc60]
>  3: (gsignal()+0x35) [0x7fab43f1dd05]
>  4: (abort()+0x186) [0x7fab43f21ab6]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fab447d46dd]
>  6: (()+0xb9926) [0x7fab447d2926]
>  7: (()+0xb9953) [0x7fab447d2953]
>  8: (()+0xb9a5e) [0x7fab447d2a5e]
>  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x396) [0x5cd9e6]
>  10: (FileJournal::committed_thru(unsigned long)+0xcd) [0x72e7cd]
>  11: (JournalingObjectStore::commit_finish()+0xb9) [0x714d79]
>  12: (FileStore::sync_entry()+0xec7) [0x70aae7]
>  13: (FileStore::SyncThread::entry()+0xd) [0x7139bd]
>  14: (()+0x6d8c) [0x7fab45993d8c]
>  15: (clone()+0x6d) [0x7fab43fd004d]
>
>
> Thread 1 (Thread 2491):
> #0  0x00007fab4599cb3b in raise () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005a77f2 in reraise_fatal (signum=6) at
> global/signal_handler.cc:59
> #2  0x00000000005a7dea in handle_fatal_signal (signum=6) at
> global/signal_handler.cc:106
> #3  <signal handler called>
> #4  0x00007fab43f1dd05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #5  0x00007fab43f21ab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #6  0x00007fab447d46dd in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #7  0x00007fab447d2926 in ?? () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> ---Type <return> to continue, or q <return> to quit---
> #8  0x00007fab447d2953 in std::terminate() () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #9  0x00007fab447d2a5e in __cxa_throw () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #10 0x00000000005cd9e6 in ceph::__ceph_assert_fail (assertion=<value
> optimized out>, file=<value optimized out>, line=<value optimized
> out>, func=<value optimized out>)
>     at common/assert.cc:70
> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
> seq=4145693) at os/FileJournal.cc:1011
> #12 0x0000000000714d79 in JournalingObjectStore::commit_finish
> (this=0x1401000) at os/JournalingObjectStore.cc:260
> #13 0x000000000070aae7 in FileStore::sync_entry (this=0x1401000) at
> os/FileStore.cc:3079
> #14 0x00000000007139bd in FileStore::SyncThread::entry (this=<value
> optimized out>) at os/FileStore.h:101
> #15 0x00007fab45993d8c in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #16 0x00007fab43fd004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #17 0x0000000000000000 in ?? ()
> (gdb)
> (gdb) thread 1
> [Switching to thread 1 (Thread 2491)]#0  0x00007fab4599cb3b in raise
> () from /lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) thread 11
> Thread ID 11 not known.
> (gdb) frame 11
> #11 0x000000000072e7cd in FileJournal::committed_thru (this=0x1410000,
> seq=4145693) at os/FileJournal.cc:1011
> 1011    os/FileJournal.cc: No such file or directory.
>         in os/FileJournal.cc
> (gdb) p seq
> $1 = 4145693
> (gdb) p last_committed_seq
> $2 = 4145768
> (gdb)
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Stratos Psomadakis
<psomas@grnet.gr>



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]