From: Olivier Bonvalet <ceph.list@daevel.fr>
To: Ilya Dryomov <ilya.dryomov@inktank.com>
Cc: Alex Elder <elder@ieee.org>,
Ceph Development <ceph-devel@vger.kernel.org>
Subject: Re: Issue #5876 : assertion failure in rbd_img_obj_callback()
Date: Tue, 25 Mar 2014 22:10:47 +0100 [thread overview]
Message-ID: <1395781847.2076.21.camel@localhost> (raw)
In-Reply-To: <1395780835.2076.15.camel@localhost>
Le mardi 25 mars 2014 à 21:53 +0100, Olivier Bonvalet a écrit :
> Le mardi 25 mars 2014 à 21:21 +0100, Olivier Bonvalet a écrit :
> > Le mardi 25 mars 2014 à 22:18 +0200, Ilya Dryomov a écrit :
> > > On Tue, Mar 25, 2014 at 9:03 PM, Alex Elder <elder@ieee.org> wrote:
> > > > On 03/25/2014 01:53 PM, Olivier Bonvalet wrote:
> > > >> Le mardi 25 mars 2014 à 12:43 -0500, Alex Elder a écrit :
> > > >>> Please try applying this, on top of the previous patch.
> > > >>> If you can then reproduce the problem we'll have a bunch
> > > >>> of new information about the particular request that's
> > > >>> leading to the failure. That might tell us what more we
> > > >>> can do to find the root cause. Thank you.
> > > >>>
> > > >>> -Alex
> > > >>>
> > > >>> PS I hope my mailer doesn't botch the long lines. It might.
> > > >>>
> > > >>
> > > >> Here the execution will continue, no more kernel panic after this
> > > >> debugging display. Is it wanted ?
> > > >
> > > >
> > > > I guess it should panic. I'm glad you mentioned this.
> > >
> > > Just in case, if you haven't done it already: stick rbd_assert(0);
> > > after the last printk in that if statement, so it looks like this:
> > >
> > > if (which != img_request->next_completion) {
> > > printk("%s: bad image object request information:\n", __func__);
> > > printk("obj_request %p\n", obj_request);
> > > printk(" ->object_name <%s>\n", obj_request->object_name);
> > > ...
> > >
> > > printk("img_request %p\n", img_request);
> > > printk(" ->snap 0x%016llx\n", img_request->snap_id);
> > > ...
> > > printk(" ->result %d\n", img_request->result);
> > >
> > > rbd_assert(0);
> > > }
> > >
> > > Thanks,
> > >
> > > Ilya
> > >
> >
> > Without the rbd_assert(0), I add this hang :
> >
> >
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255933] rbd_img_obj_callback: bad image object request information:
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255938] obj_request ffff88025a2b3c48
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255940] ->object_name <rb.0.1536881.238e1f29.000000000439>
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255941] ->offset 0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255943] ->length 28672
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255944] ->type 0x1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255945] ->flags 0x3
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255946] ->which 1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255948] ->xferred 28672
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255949] ->result 0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255950] img_request ffff8802536c4a60
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255952] ->snap 0xffff880257f85ec0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255953] ->offset 4534026240
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255954] ->length 45056
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255955] ->flags 0x1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255957] ->obj_request_count 1
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255958] ->next_completion 2
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255959] ->xferred 45056
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255960] ->result 0
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962]
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962] Assertion failure in rbd_img_obj_callback() at line 2162:
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962]
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962] rbd_assert(which < img_request->obj_request_count);
> > Mar 25 21:17:58 murmillia kernel: [ 2205.255962]
> > Mar 25 21:17:58 murmillia kernel: [ 2205.256141] ------------[ cut here ]------------
> > Mar 25 21:17:58 murmillia kernel: [ 2205.256178] kernel BUG at drivers/block/rbd.c:2162!
> >
> >
> > --
>
> An other one :
>
> Mar 25 21:52:50 alg kernel: [ 1781.377690] rbd_img_obj_callback: bad image object request information:
> Mar 25 21:52:50 alg kernel: [ 1781.377696] obj_request ffff88021dda2ae8
> Mar 25 21:52:50 alg kernel: [ 1781.377698] ->object_name <(null)>
> Mar 25 21:52:50 alg kernel: [ 1781.377699] ->offset 0
> Mar 25 21:52:50 alg kernel: [ 1781.377701] ->length 12288
> Mar 25 21:52:50 alg kernel: [ 1781.377702] ->type 0x1
> Mar 25 21:52:50 alg kernel: [ 1781.377703] ->flags 0x3
> Mar 25 21:52:50 alg kernel: [ 1781.377704] ->which 4294967295
> Mar 25 21:52:50 alg kernel: [ 1781.377705] ->xferred 12288
> Mar 25 21:52:50 alg kernel: [ 1781.377706] ->result 0
> Mar 25 21:52:50 alg kernel: [ 1781.377707] img_request ffff880223f396a0
> Mar 25 21:52:50 alg kernel: [ 1781.377709] ->snap 0xffff880231dd8cc0
> Mar 25 21:52:50 alg kernel: [ 1781.377710] ->offset 1119846400
> Mar 25 21:52:50 alg kernel: [ 1781.377711] ->length 45056
> Mar 25 21:52:50 alg kernel: [ 1781.377712] ->flags 0x1
> Mar 25 21:52:50 alg kernel: [ 1781.377713] ->obj_request_count 0
> Mar 25 21:52:50 alg kernel: [ 1781.377713] ->next_completion 2
> Mar 25 21:52:50 alg kernel: [ 1781.377714] ->xferred 45056
> Mar 25 21:52:50 alg kernel: [ 1781.377715] ->result 0
> Mar 25 21:52:50 alg kernel: [ 1781.377717]
> Mar 25 21:52:50 alg kernel: [ 1781.377717] Assertion failure in rbd_img_obj_callback() at line 2162:
> Mar 25 21:52:50 alg kernel: [ 1781.377717]
> Mar 25 21:52:50 alg kernel: [ 1781.377717] rbd_assert(which < img_request->obj_request_count);
> Mar 25 21:52:50 alg kernel: [ 1781.377717]
> Mar 25 21:52:50 alg kernel: [ 1781.377859] ------------[ cut here ]------------
>
>
> --
The third (now with rbd_assort(0)) :
Mar 25 22:08:12 alg kernel: [ 598.301895] rbd_img_obj_callback: bad image object request information:
Mar 25 22:08:12 alg kernel: [ 598.301900] obj_request ffff88022409e1b8
Mar 25 22:08:12 alg kernel: [ 598.301901] ->object_name <(null)>
Mar 25 22:08:12 alg kernel: [ 598.301902] ->offset 0
Mar 25 22:08:12 alg kernel: [ 598.301903] ->length 8192
Mar 25 22:08:12 alg kernel: [ 598.301904] ->type 0x1
Mar 25 22:08:12 alg kernel: [ 598.301905] ->flags 0x3
Mar 25 22:08:12 alg kernel: [ 598.301906] ->which 4294967295
Mar 25 22:08:12 alg kernel: [ 598.301906] ->xferred 8192
Mar 25 22:08:12 alg kernel: [ 598.301907] ->result 0
Mar 25 22:08:12 alg kernel: [ 598.301908] img_request ffff8802303bff10
Mar 25 22:08:12 alg kernel: [ 598.301909] ->snap 0xffff88022711f500
Mar 25 22:08:12 alg kernel: [ 598.301910] ->offset 4492079104
Mar 25 22:08:12 alg kernel: [ 598.301911] ->length 28672
Mar 25 22:08:12 alg kernel: [ 598.301912] ->flags 0x1
Mar 25 22:08:12 alg kernel: [ 598.301913] ->obj_request_count 0
Mar 25 22:08:12 alg kernel: [ 598.301913] ->next_completion 2
Mar 25 22:08:12 alg kernel: [ 598.301914] ->xferred 28672
Mar 25 22:08:12 alg kernel: [ 598.301915] ->result 0
Mar 25 22:08:12 alg kernel: [ 598.301916]
Mar 25 22:08:12 alg kernel: [ 598.301916] Assertion failure in rbd_img_obj_callback() at line 2159:
Mar 25 22:08:12 alg kernel: [ 598.301916]
Mar 25 22:08:12 alg kernel: [ 598.301916] rbd_assert(0);
Mar 25 22:08:12 alg kernel: [ 598.301916]
Mar 25 22:08:12 alg kernel: [ 598.302093] ------------[ cut here ]------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2014-03-25 21:10 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-25 8:39 Issue #5876 : assertion failure in rbd_img_obj_callback() Olivier Bonvalet
2014-03-25 9:04 ` Ilya Dryomov
[not found] ` <1395739214.2823.34.camel@localhost>
2014-03-25 9:52 ` Ilya Dryomov
2014-03-25 11:48 ` Alex Elder
2014-03-25 12:34 ` Ilya Dryomov
2014-03-25 12:51 ` Alex Elder
2014-03-25 12:57 ` Ilya Dryomov
2014-03-25 13:18 ` Olivier Bonvalet
2014-03-25 13:29 ` Alex Elder
2014-03-25 13:31 ` Alex Elder
2014-03-25 14:01 ` Olivier Bonvalet
2014-03-25 17:15 ` Olivier Bonvalet
2014-03-25 17:21 ` Alex Elder
2014-03-25 18:53 ` Olivier Bonvalet
2014-03-25 17:43 ` Alex Elder
2014-03-25 18:53 ` Olivier Bonvalet
2014-03-25 19:03 ` Alex Elder
2014-03-25 20:18 ` Ilya Dryomov
2014-03-25 20:21 ` Olivier Bonvalet
2014-03-25 20:24 ` Alex Elder
2014-03-25 20:29 ` Olivier Bonvalet
2014-03-25 20:44 ` Alex Elder
2014-03-25 21:03 ` Olivier Bonvalet
2014-03-25 20:41 ` Alex Elder
2014-03-25 20:53 ` Olivier Bonvalet
2014-03-25 21:10 ` Olivier Bonvalet [this message]
2014-03-25 21:20 ` Ilya Dryomov
[not found] ` <1395782577.2076.23.camel@localhost>
2014-03-25 21:25 ` Ilya Dryomov
2014-03-25 21:41 ` Olivier Bonvalet
2014-03-25 21:49 ` Ilya Dryomov
2014-03-25 21:54 ` Olivier Bonvalet
2014-03-25 22:17 ` Olivier Bonvalet
2014-03-25 22:46 ` Alex Elder
2014-03-25 23:04 ` Olivier Bonvalet
2014-03-26 0:00 ` Alex Elder
2014-03-26 1:33 ` Olivier Bonvalet
2014-03-26 1:50 ` Olivier Bonvalet
2014-03-26 1:55 ` Alex Elder
2014-03-26 2:40 ` Olivier Bonvalet
2014-03-26 2:42 ` Alex Elder
2014-03-26 2:45 ` Olivier Bonvalet
2014-03-26 3:54 ` Alex Elder
2014-03-26 4:00 ` Olivier Bonvalet
2014-03-26 5:00 ` Alex Elder
2014-03-26 11:13 ` Alex Elder
2014-03-26 11:43 ` Ilya Dryomov
2014-03-26 11:47 ` Alex Elder
2014-03-26 12:05 ` Ilya Dryomov
2014-03-26 20:58 ` Alex Elder
2014-03-27 7:48 ` Olivier Bonvalet
2014-03-27 8:45 ` Ilya Dryomov
2014-03-27 8:49 ` Olivier Bonvalet
2014-03-26 2:35 ` Olivier Bonvalet
2014-03-26 2:54 ` Alex Elder
2014-03-26 3:58 ` Olivier Bonvalet
2014-04-05 1:16 ` Olivier Bonvalet
2014-04-05 1:57 ` Alex Elder
2014-04-05 8:09 ` Olivier Bonvalet
2014-04-05 13:08 ` Alex Elder
2014-04-25 11:37 ` Olivier Bonvalet
2014-04-25 12:17 ` Alex Elder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1395781847.2076.21.camel@localhost \
--to=ceph.list@daevel.fr \
--cc=ceph-devel@vger.kernel.org \
--cc=elder@ieee.org \
--cc=ilya.dryomov@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.