From: Greg Harris <greg.harris@metacarta.com>
To: Jeremy Fitzhardinge <jeremy@goop.org>
Cc: Jens Axboe <jens.axboe@oracle.com>,
xen-devel <xen-devel@lists.xensource.com>
Subject: Re: Kernel Panic in xen-blkfront.c:blkif_queue_request under 2.6.28
Date: Tue, 3 Feb 2009 15:37:29 -0500 (EST) [thread overview]
Message-ID: <18316821.8169281233693449109.JavaMail.root@ouachita> (raw)
In-Reply-To: <4987821C.9040605@goop.org>
After applying the patch we were able to reproduce the panic and the additional debugging output is attached. The driver appears to re-try the request several times before dying:
Writing inode tables: ------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff80232ae3>] enqueue_task+0x13/0x30
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff8036fb35>] blk_invoke_request_fn+0xa5/0x110
[<ffffffff80407a08>] kick_pending_request_queues+0x18/0x30
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x130
[<ffffffff80246217>] __do_softirq+0xe7/0x180
[<ffffffff8059c65e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff8020de8c>] xen_safe_halt+0xc/0x20
[<ffffffff8020c1fa>] xen_idle+0x2a/0x50
[<ffffffff80210041>] cpu_idle+0x41/0x70
---[ end trace 107c74ebf2b50a63 ]---
METACARTA: too many segments for ring (11): req->nr_phys_segments = 11
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3072 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3584 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 0 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 512 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1024 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 3072 len 512
------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x130
[<ffffffff80246217>] __do_softirq+0xe7/0x180
[<ffffffff8059c65e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff8020de8c>] xen_safe_halt+0xc/0x20
[<ffffffff8020c1fa>] xen_idle+0x2a/0x50
[<ffffffff80210041>] cpu_idle+0x41/0x70
---[ end trace 107c74ebf2b50a63 ]---
METACARTA: too many segments for ring (11): req->nr_phys_segments = 11
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3072 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3584 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 0 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 512 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1024 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 3072 len 512
------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff80232ae3>] enqueue_task+0x13/0x30
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff8036fb35>] blk_invoke_request_fn+0xa5/0x110
[<ffffffff80407a08>] kick_pending_request_queues+0x18/0x30
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x130
[<ffffffff80246217>] __do_softirq+0xe7/0x180
[<ffffffff8059c65e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff8020de8c>] xen_safe_halt+0xc/0x20
[<ffffffff8020c1fa>] xen_idle+0x2a/0x50
[<ffffffff80210041>] cpu_idle+0x41/0x70
---[ end trace 107c74ebf2b50a63 ]---
METACARTA: too many segments for ring (11): req->nr_phys_segments = 11
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3072 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3584 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 0 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 512 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1024 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 3072 len 512
------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x13
We also attempted changing the blk_queue_max_*_segments calls to use BLKIF_MAX_SEGMENTS_PER_REQUEST - 1 and our spinner was able to run overnight without any panics...
---
Greg Harris
System Administrator
MetaCarta, Inc.
(O) +1 (617) 301-5530
(M) +1 (781) 258-4474
----- "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
> Jens Axboe wrote:
> > To shed some more light on this, I'd suggest changing that BUG_ON()
> to
> > some code that simply dumps each segment (each bvec in the iterator
> > list) from start to finish along with values of
> > request->nr_phys_segments and size info.
> >
>
> OK, something like this?
>
> J
>
> Subject: xen/blkfront: try to track down over-segment BUG_ON in
> blkfront
>
> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> ---
> drivers/block/xen-blkfront.c | 24 +++++++++++++++++++++++-
> 1 file changed, 23 insertions(+), 1 deletion(-)
>
> ===================================================================
> --- a/drivers/block/xen-blkfront.c
> +++ b/drivers/block/xen-blkfront.c
> @@ -240,7 +240,10 @@
>
> ring_req->nr_segments = 0;
> rq_for_each_segment(bvec, req, iter) {
> - BUG_ON(ring_req->nr_segments == BLKIF_MAX_SEGMENTS_PER_REQUEST);
> + if (WARN_ON(ring_req->nr_segments >=
> + BLKIF_MAX_SEGMENTS_PER_REQUEST))
> + goto dump_req;
> +
> buffer_mfn = pfn_to_mfn(page_to_pfn(bvec->bv_page));
> fsect = bvec->bv_offset >> 9;
> lsect = fsect + (bvec->bv_len >> 9) - 1;
> @@ -274,6 +277,25 @@
> gnttab_free_grant_references(gref_head);
>
> return 0;
> +
> +dump_req:
> + {
> + int i;
> +
> + printk(KERN_DEBUG "too many segments for ring (%d): "
> + "req->nr_phys_segments = %d\n",
> + BLKIF_MAX_SEGMENTS_PER_REQUEST, req->nr_phys_segments);
> +
> + i = 0;
> + rq_for_each_segment(bvec, req, iter) {
> + printk(KERN_DEBUG
> + " %d: bio page %p pfn %lx off %u len %u\n",
> + i++, bvec->bv_page, page_to_pfn(bvec->bv_page),
> + bvec->bv_offset, bvec->bv_len);
> + }
> + }
> +
> + return 1;
> }
next prev parent reply other threads:[~2009-02-03 20:37 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <11750686.8060601233586380769.JavaMail.root@ouachita>
2009-02-02 14:53 ` Kernel Panic in xen-blkfront.c:blkif_queue_request under 2.6.28 Greg Harris
[not found] ` <20090202161425.GK30821@kernel.dk>
2009-02-02 23:30 ` Jeremy Fitzhardinge
2009-02-03 20:37 ` Greg Harris [this message]
2009-02-04 16:50 ` Jeremy Fitzhardinge
[not found] <24523950.8056631233583862811.JavaMail.root@ouachita>
2009-02-02 14:11 ` Greg Harris
2009-01-30 14:50 Greg Harris
2009-02-02 6:19 ` Jeremy Fitzhardinge
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=18316821.8169281233693449109.JavaMail.root@ouachita \
--to=greg.harris@metacarta.com \
--cc=jens.axboe@oracle.com \
--cc=jeremy@goop.org \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.