jewel: rbd: FAILED assert(m_in_flight_object_closes > 0)

[Loic Dachary <loic@dachary.org>]

Hi Jason,

The 

rbd/qemu/{cache/writeback.yaml cachepool/ec-cache.yaml clusters/{fixed-3.yaml openstack.yaml} features/journaling.yaml fs/xfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}

job failed twice in a row

http://pulpito.ceph.com/loic-2016-10-14_06:48:18-rbd-jewel-backports---basic-smithi/473422
http://pulpito.ceph.com/loic-2016-10-13_09:17:31-rbd-jewel-backports---basic-smithi/470997

with

2016-10-14T06:57:57.700 INFO:tasks.qemu.client.0.smithi087.stdout:mount: block device /dev/sr0 is write-protected, mounting read-only
2016-10-14T06:59:18.374 INFO:tasks.ceph.osd.5.smithi096.stderr:2016-10-14 06:59:18.373238 7f89520fb700 -1 reset not still connected to 0x7f89737050c0
2016-10-14T06:59:58.946 INFO:tasks.ceph.osd.5.smithi096.stderr:2016-10-14 06:59:58.946646 7f89520fb700 -1 reset not still connected to 0x7f8973913180
2016-10-14T07:07:59.122 INFO:tasks.qemu.client.0.smithi087.stderr:main-loop: WARNING: I/O thread spun for 1000 iterations
2016-10-14T07:09:11.712 INFO:tasks.ceph.osd.5.smithi096.stderr:2016-10-14 07:09:11.711124 7f89520fb700 -1 reset not still connected to 0x7f8973913180
2016-10-14T07:10:28.034 INFO:tasks.ceph.osd.5.smithi096.stderr:2016-10-14 07:10:28.034047 7f89520fb700 -1 reset not still connected to 0x7f8973913180
2016-10-14T07:14:20.062 INFO:tasks.qemu.client.0.smithi087.stderr:journal/JournalRecorder.cc: In function 'void journal::JournalRecorder::handle_closed(journal::ObjectRecorder*)' thread 7fe0b3066700 time 2016-10-14 07:14:20.069199
2016-10-14T07:14:20.063 INFO:tasks.qemu.client.0.smithi087.stderr:journal/JournalRecorder.cc: 321: FAILED assert(m_in_flight_object_closes > 0)
2016-10-14T07:14:20.091 INFO:tasks.qemu.client.0.smithi087.stderr: ceph version 10.2.3-266-g94a5296 (94a52962cdb0eb3d2f2eba11daa0e14ae349734e)
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 1: (()+0x29e865) [0x7fe0cf2c1865]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 2: (()+0x1a7589) [0x7fe0cf1ca589]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 3: (()+0x1b2c46) [0x7fe0cf1d5c46]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 4: (()+0x1b5b23) [0x7fe0cf1d8b23]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 5: (()+0x75ef9) [0x7fe0cf098ef9]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 6: (()+0xc1487) [0x7fe0cf0e4487]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 7: (()+0xc15a2) [0x7fe0cf0e45a2]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 8: (()+0xb1c39) [0x7fe0cf0d4c39]
2016-10-14T07:14:20.092 INFO:tasks.qemu.client.0.smithi087.stderr: 9: (()+0x9ebad) [0x7fe0c5941bad]
2016-10-14T07:14:20.093 INFO:tasks.qemu.client.0.smithi087.stderr: 10: (()+0x87099) [0x7fe0c592a099]
2016-10-14T07:14:20.093 INFO:tasks.qemu.client.0.smithi087.stderr: 11: (()+0x173d06) [0x7fe0c5a16d06]
2016-10-14T07:14:20.093 INFO:tasks.qemu.client.0.smithi087.stderr: 12: (()+0x7dc5) [0x7fe0d8d8cdc5]
2016-10-14T07:14:20.093 INFO:tasks.qemu.client.0.smithi087.stderr: 13: (clone()+0x6d) [0x7fe0c2fb5ced]
2016-10-14T07:14:20.093 INFO:tasks.qemu.client.0.smithi087.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-10-14T07:14:23.511 INFO:tasks.qemu.client.0.smithi087.stderr:daemon-helper: command crashed with signal 6
2016-10-14T07:14:23.518 INFO:tasks.rbd:Deleting rbd clones...

It used the jewel-backports branch ( http://tracker.ceph.com/issues/17487#note-9 ).

I looked in the tracker for m_in_flight_object_closes and did not find any match. Since https://github.com/ceph/ceph/pull/11433 has many journal related commits, I wonder if maybe something is missing ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre