From: Olivier Bonvalet <ceph.list@daevel.fr>
To: Ilya Dryomov <ilya.dryomov@inktank.com>
Cc: Alex Elder <elder@ieee.org>,
Ceph Development <ceph-devel@vger.kernel.org>
Subject: Re: Issue #5876 : assertion failure in rbd_img_obj_callback()
Date: Tue, 25 Mar 2014 22:10:47 +0100 [thread overview]
Message-ID: <1395781847.2076.21.camel@localhost> (raw)
In-Reply-To: <1395780835.2076.15.camel@localhost>
Le mardi 25 mars 2014 à 21:53 +0100, Olivier Bonvalet a écrit :
> Le mardi 25 mars 2014 à 21:21 +0100, Olivier Bonvalet a écrit :
> > Le mardi 25 mars 2014 à 22:18 +0200, Ilya Dryomov a écrit :
> > > On Tue, Mar 25, 2014 at 9:03 PM, Alex Elder <elder@ieee.org> wrote:
> > > > On 03/25/2014 01:53 PM, Olivier Bonvalet wrote:
> > > >> Le mardi 25 mars 2014 à 12:43 -0500, Alex Elder a écrit :
> > > >>> Please try applying this, on top of the previous patch.
> > > >>> If you can then reproduce the problem we'll have a bunch
> > > >>> of new information about the particular request that's
> > > >>> leading to the failure. That might tell us what more we
> > > >>> can do to find the root cause. Thank you.
> > > >>>
> > > >>> -Alex
> > > >>>
> > > >>> PS I hope my mailer doesn't botch the long lines. It might.
> > > >>>
> > > >>
> > > >> Here the execution will continue, no more kernel panic after this
> > > >> debugging display. Is it wanted ?
> > > >
> > > >
> > > > I guess it should panic. I'm glad you mentioned this.
> > >
> > > Just in case, if you haven't done it already: stick rbd_assert(0);
> > > after the last printk in that if statement, so it looks like this:
> > >
> > > if (which != img_request->next_completion) {
> > > printk("%s: bad image object request information:\n", __func__);
> > > printk("obj_request %p\n", obj_request);
> > > printk(" ->object_name <%s>\n", obj_request->object_name);
> > > ...
> > >
> > > printk("img_request %p\n", img_request);
> > > printk(" ->snap 0x%016llx\n", img_request->snap_id);
> > > ...
> > > printk(" ->result %d\n", img_request->result);
> > >
> > > rbd_assert(0);
> > > }
> > >
> > > Thanks,
> > >
> > > Ilya
> > >
> >
> > Without the rbd_assert(0), I add this hang :
> >
> >
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255933] rbd_img_obj_callback: bad image object request information:
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255938] obj_request ffff88025a2b3c48
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255940] ->object_name <rb.0.1536881.238e1f29.000000000439>
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255941] ->offset 0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255943] ->length 28672
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255944] ->type 0x1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255945] ->flags 0x3
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255946] ->which 1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255948] ->xferred 28672
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255949] ->result 0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255950] img_request ffff8802536c4a60
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255952] ->snap 0xffff880257f85ec0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255953] ->offset 4534026240
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255954] ->length 45056
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255955] ->flags 0x1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255957] ->obj_request_count 1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255958] ->next_completion 2
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255959] ->xferred 45056
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255960] ->result 0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962]
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962] Assertion failure in rbd_img_obj_callback() at line 2162:
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962]
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962] rbd_assert(which < img_request->obj_request_count);
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962]
> > Mar 25 21:17:58 murmillia kernel: [ 2205.256141] ------------[ cut here ]------------
> > Mar 25 21:17:58 murmillia kernel: [ 2205.256178] kernel BUG at drivers/block/rbd.c:2162!
> >
> >
> > --
>
> An other one :
>
> Mar 25 21:52:50 alg kernel: [ 1781.377690] rbd_img_obj_callback: bad image object request information:
> Mar 25 21:52:50 alg kernel: [ 1781.377696] obj_request ffff88021dda2ae8
> Mar 25 21:52:50 alg kernel: [ 1781.377698] ->object_name <(null)>
> Mar 25 21:52:50 alg kernel: [ 1781.377699] ->offset 0
> Mar 25 21:52:50 alg kernel: [ 1781.377701] ->length 12288
> Mar 25 21:52:50 alg kernel: [ 1781.377702] ->type 0x1
> Mar 25 21:52:50 alg kernel: [ 1781.377703] ->flags 0x3
> Mar 25 21:52:50 alg kernel: [ 1781.377704] ->which 4294967295
> Mar 25 21:52:50 alg kernel: [ 1781.377705] ->xferred 12288
> Mar 25 21:52:50 alg kernel: [ 1781.377706] ->result 0
> Mar 25 21:52:50 alg kernel: [ 1781.377707] img_request ffff880223f396a0
> Mar 25 21:52:50 alg kernel: [ 1781.377709] ->snap 0xffff880231dd8cc0
> Mar 25 21:52:50 alg kernel: [ 1781.377710] ->offset 1119846400
> Mar 25 21:52:50 alg kernel: [ 1781.377711] ->length 45056
> Mar 25 21:52:50 alg kernel: [ 1781.377712] ->flags 0x1
> Mar 25 21:52:50 alg kernel: [ 1781.377713] ->obj_request_count 0
> Mar 25 21:52:50 alg kernel: [ 1781.377713] ->next_completion 2
> Mar 25 21:52:50 alg kernel: [ 1781.377714] ->xferred 45056
> Mar 25 21:52:50 alg kernel: [ 1781.377715] ->result 0
> Mar 25 21:52:50 alg kernel: [ 1781.377717]
> Mar 25 21:52:50 alg kernel: [ 1781.377717] Assertion failure in rbd_img_obj_callback() at line 2162:
> Mar 25 21:52:50 alg kernel: [ 1781.377717]
> Mar 25 21:52:50 alg kernel: [ 1781.377717] rbd_assert(which < img_request->obj_request_count);
> Mar 25 21:52:50 alg kernel: [ 1781.377717]
> Mar 25 21:52:50 alg kernel: [ 1781.377859] ------------[ cut here ]------------
>
>
> --
The third (now with rbd_assort(0)) :
Mar 25 22:08:12 alg kernel: [ 598.301895] rbd_img_obj_callback: bad image object request information:
Mar 25 22:08:12 alg kernel: [ 598.301900] obj_request ffff88022409e1b8
Mar 25 22:08:12 alg kernel: [ 598.301901] ->object_name <(null)>
Mar 25 22:08:12 alg kernel: [ 598.301902] ->offset 0
Mar 25 22:08:12 alg kernel: [ 598.301903] ->length 8192
Mar 25 22:08:12 alg kernel: [ 598.301904] ->type 0x1
Mar 25 22:08:12 alg kernel: [ 598.301905] ->flags 0x3
Mar 25 22:08:12 alg kernel: [ 598.301906] ->which 4294967295
Mar 25 22:08:12 alg kernel: [ 598.301906] ->xferred 8192
Mar 25 22:08:12 alg kernel: [ 598.301907] ->result 0
Mar 25 22:08:12 alg kernel: [ 598.301908] img_request ffff8802303bff10
Mar 25 22:08:12 alg kernel: [ 598.301909] ->snap 0xffff88022711f500
Mar 25 22:08:12 alg kernel: [ 598.301910] ->offset 4492079104
Mar 25 22:08:12 alg kernel: [ 598.301911] ->length 28672
Mar 25 22:08:12 alg kernel: [ 598.301912] ->flags 0x1
Mar 25 22:08:12 alg kernel: [ 598.301913] ->obj_request_count 0
Mar 25 22:08:12 alg kernel: [ 598.301913] ->next_completion 2
Mar 25 22:08:12 alg kernel: [ 598.301914] ->xferred 28672
Mar 25 22:08:12 alg kernel: [ 598.301915] ->result 0
Mar 25 22:08:12 alg kernel: [ 598.301916]
Mar 25 22:08:12 alg kernel: [ 598.301916] Assertion failure in rbd_img_obj_callback() at line 2159:
Mar 25 22:08:12 alg kernel: [ 598.301916]
Mar 25 22:08:12 alg kernel: [ 598.301916] rbd_assert(0);
Mar 25 22:08:12 alg kernel: [ 598.301916]
Mar 25 22:08:12 alg kernel: [ 598.302093] ------------[ cut here ]------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2014-03-25 21:10 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-25 8:39 Issue #5876 : assertion failure in rbd_img_obj_callback() Olivier Bonvalet
2014-03-25 9:04 ` Ilya Dryomov
[not found] ` <1395739214.2823.34.camel@localhost>
2014-03-25 9:52 ` Ilya Dryomov
2014-03-25 11:48 ` Alex Elder
2014-03-25 12:34 ` Ilya Dryomov
2014-03-25 12:51 ` Alex Elder
2014-03-25 12:57 ` Ilya Dryomov
2014-03-25 13:18 ` Olivier Bonvalet
2014-03-25 13:29 ` Alex Elder
2014-03-25 13:31 ` Alex Elder
2014-03-25 14:01 ` Olivier Bonvalet
2014-03-25 17:15 ` Olivier Bonvalet
2014-03-25 17:21 ` Alex Elder
2014-03-25 18:53 ` Olivier Bonvalet
2014-03-25 17:43 ` Alex Elder
2014-03-25 18:53 ` Olivier Bonvalet
2014-03-25 19:03 ` Alex Elder
2014-03-25 20:18 ` Ilya Dryomov
2014-03-25 20:21 ` Olivier Bonvalet
2014-03-25 20:24 ` Alex Elder
2014-03-25 20:29 ` Olivier Bonvalet
2014-03-25 20:44 ` Alex Elder
2014-03-25 21:03 ` Olivier Bonvalet
2014-03-25 20:41 ` Alex Elder
2014-03-25 20:53 ` Olivier Bonvalet
2014-03-25 21:10 ` Olivier Bonvalet [this message]
2014-03-25 21:20 ` Ilya Dryomov
[not found] ` <1395782577.2076.23.camel@localhost>
2014-03-25 21:25 ` Ilya Dryomov
2014-03-25 21:41 ` Olivier Bonvalet
2014-03-25 21:49 ` Ilya Dryomov
2014-03-25 21:54 ` Olivier Bonvalet
2014-03-25 22:17 ` Olivier Bonvalet
2014-03-25 22:46 ` Alex Elder
2014-03-25 23:04 ` Olivier Bonvalet
2014-03-26 0:00 ` Alex Elder
2014-03-26 1:33 ` Olivier Bonvalet
2014-03-26 1:50 ` Olivier Bonvalet
2014-03-26 1:55 ` Alex Elder
2014-03-26 2:40 ` Olivier Bonvalet
2014-03-26 2:42 ` Alex Elder
2014-03-26 2:45 ` Olivier Bonvalet
2014-03-26 3:54 ` Alex Elder
2014-03-26 4:00 ` Olivier Bonvalet
2014-03-26 5:00 ` Alex Elder
2014-03-26 11:13 ` Alex Elder
2014-03-26 11:43 ` Ilya Dryomov
2014-03-26 11:47 ` Alex Elder
2014-03-26 12:05 ` Ilya Dryomov
2014-03-26 20:58 ` Alex Elder
2014-03-27 7:48 ` Olivier Bonvalet
2014-03-27 8:45 ` Ilya Dryomov
2014-03-27 8:49 ` Olivier Bonvalet
2014-03-26 2:35 ` Olivier Bonvalet
2014-03-26 2:54 ` Alex Elder
2014-03-26 3:58 ` Olivier Bonvalet
2014-04-05 1:16 ` Olivier Bonvalet
2014-04-05 1:57 ` Alex Elder
2014-04-05 8:09 ` Olivier Bonvalet
2014-04-05 13:08 ` Alex Elder
2014-04-25 11:37 ` Olivier Bonvalet
2014-04-25 12:17 ` Alex Elder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1395781847.2076.21.camel@localhost \
--to=ceph.list@daevel.fr \
--cc=ceph-devel@vger.kernel.org \
--cc=elder@ieee.org \
--cc=ilya.dryomov@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox