From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Kernel Panic in xen-blkfront.c:blkif_queue_request under 2.6.28 Date: Sun, 01 Feb 2009 22:19:56 -0800 Message-ID: <4986908C.8050204@goop.org> References: <2411544.7918341233327005412.JavaMail.root@ouachita> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <2411544.7918341233327005412.JavaMail.root@ouachita> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Greg Harris Cc: xen-devel , Jens Axboe List-Id: xen-devel@lists.xenproject.org Greg Harris wrote: > Hi, > > I've run into several panics around the Xen block frontend driver in 2.6.28. It appears that when the kernel issues block queue requests in blkif_queue_request the number of segments exceeds BLKIF_MAX_SEGMENTS_PER_REQUEST triggering the panic despite the call to set the maximum number of segments during the queue initialization (xlvdb_init_blk_queue calls blk_queue_max_phys_segments and blk_queue_max_hw_segments with BLKIF_MAX_SEGMENTS_PER_REQUEST as parameters). > I've got a few reports if this, but I haven't managed to reproduce it myself. But from this description it sounds like the problem is in the upper layers presenting too many segments, rather than a bug in the block driver itself. Jens, does that sound likely? Thanks, J > Attached are two panics: > > kernel BUG at drivers/block/xen-blkfront.c:243! > invalid opcode: 0000 [#1] SMP > last sysfs file: /sys/block/xvda/dev > CPU 0 > Modules linked in: > Pid: 0, comm: swapper Not tainted 2.6.28-metacarta-appliance-1 #2 > RIP: e030:[] [] > do_blkif_request+0x2f0/0x380 > RSP: e02b:ffffffff80865dd8 EFLAGS: 00010046 > RAX: 0000000000000000 RBX: ffff880366ee33c0 RCX: ffff880366ee33c0 > RDX: ffff880366f15d90 RSI: 000000000000000a RDI: 0000000000000303 > RBP: ffff88039d78b190 R08: 0000000000001818 R09: ffff88038fb7a9e0 > R10: 0000000000000004 R11: 000000000000001a R12: 0000000000000303 > R13: 0000000000000001 R14: ffff880366f15da0 R15: 0000000000000000 > FS: 0000000000000000(0000) GS:ffffffff807a1980(0000) knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00000000f7f54444 CR3: 00000003977e5000 CR4: 0000000000002620 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process swapper (pid: 0, threadinfo ffffffff807a6000, task ffffffff806f0360) > Stack: > 000000000000004c ffff88038fb7a9e0 ffff88039a5f4000 ffff880366f123b8 > 0000000680298eec 000000000000000f ffff88039a5f4000 0000000066edc808 > ffff880366ee33c0 ffff88038fb7aa00 ffffffff00000001 ffff88038fb7a9e0 > Call Trace: > <0> [] ? blk_invoke_request_fn+0xa5/0x110 > [] ? kick_pending_request_queues+0x18/0x30 > [] ? blkif_interrupt+0x197/0x1e0 > [] ? handle_IRQ_event+0x39/0x80 > [] ? handle_level_irq+0x96/0x120 > [] ? do_IRQ+0x85/0x110 > [] ? xen_evtchn_do_upcall+0xe5/0x130 > [] ? __do_softirq+0xe7/0x180 > [] ? xen_do_hypervisor_callback+0x1e/0x30 > <0> [] ? _stext+0x3aa/0x1000 > [] ? _stext+0x3aa/0x1000 > [] ? xen_safe_halt+0xc/0x20 > [] ? xen_idle+0x2a/0x50 > [] ? cpu_idle+0x41/0x70 > Code: fa d0 00 00 00 48 8d bc 07 88 00 00 00 e8 b9 dd f7 ff 8b 7c 24 54 e8 90 fb > fb ff ff 44 24 24 e9 3b fd ff ff 0f 0b eb fe 66 66 90 <0f> 0b eb fe 48 8b 7c 24 > 30 48 8b 54 24 30 b9 0b 00 00 00 48 c7 > RIP [] do_blkif_request+0x2f0/0x380 > RSP > Kernel panic - not syncing: Fatal exception in interrupt > > kernel BUG at drivers/block/xen-blkfront.c:243! > invalid opcode: 0000 [#1] SMP > last sysfs file: /sys/block/xvda/dev > CPU 0 > Modules linked in: > Pid: 0, comm: swapper Not tainted 2.6.28-metacarta-appliance-1 #2 > RIP: e030:[] [] > do_blkif_request+0x2f0/0x380 > RSP: e02b:ffffffff80865dd8 EFLAGS: 00010046 > RAX: 0000000000000000 RBX: ffff880366f2a9c0 RCX: ffff880366f2a9c0 > RDX: ffff880366f233b0 RSI: 000000000000000a RDI: 0000000000000168 > RBP: ffff88039d895cf0 R08: 0000000000000b40 R09: ffff88038fb029e0 > R10: 000000000000000f R11: 000000000000001a R12: 0000000000000168 > R13: 0000000000000001 R14: ffff880366f233c0 R15: 0000000000000000 > FS: 0000000000000000(0000) GS:ffffffff807a1980(0000) knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00000000f7f9c444 CR3: 000000039e7ea000 CR4: 0000000000002620 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process swapper (pid: 0, threadinfo ffffffff807a6000, task ffffffff806f0360) > Stack: > 000000000000004c ffff88038fb029e0 ffff88039a5d8000 ffff880366f11938 > 0000000980298eec 000000000000001d ffff88039a5d8000 000004008036f3da > ffff880366f2a9c0 ffff88038fb02a00 ffffffff00000001 ffff88038fb029e0 > Call Trace: > <0> [] ? blk_invoke_request_fn+0xa5/0x110 > [] ? kick_pending_request_queues+0x18/0x30 > [] ? blkif_interrupt+0x197/0x1e0 > [] ? handle_IRQ_event+0x39/0x80 > [] ? handle_level_irq+0x96/0x120 > [] ? do_IRQ+0x85/0x110 > [] ? xen_evtchn_do_upcall+0xe5/0x130 > [] ? __do_softirq+0xe7/0x180 > [] ? xen_do_hypervisor_callback+0x1e/0x30 > <0> [] ? _stext+0x3aa/0x1000 > [] ? _stext+0x3aa/0x1000 > [] ? xen_safe_halt+0xc/0x20 > [] ? xen_idle+0x2a/0x50 > [] ? cpu_idle+0x41/0x70 > Code: fa d0 00 00 00 48 8d bc 07 88 00 00 00 e8 b9 dd f7 ff 8b 7c 24 54 e8 90 fb > fb ff ff 44 24 24 e9 3b fd ff ff 0f 0b eb fe 66 66 90 <0f> 0b eb > fe 48 8b 7c 24 30 48 8b 54 24 30 b9 0b 00 00 00 48 c7 > RIP [] do_blkif_request+0x2f0/0x380 > RSP > Kernel panic - not syncing: Fatal exception in interrupt > > We've encountered the a similar panic using Xen 3.2.1 (debian-backports, 2.6.18-6-xen-amd64 kernel) and Xen 3.2.0 (Ubuntu Hardy, 2.6.24-23-xen kernel) running in para-virtual mode. The source around the line referenced in the panic is: > > rq_for_each_segment(bvec, req, iter) { > BUG_ON(ring_req->nr_segments == BLKIF_MAX_SEGMENTS_PER_REQUEST); > ... > handle the segment > ... > ring_req->nr_segments++; > } > > I'm able to reliably reproduce this panic through a certain workload (usually through creating file-systems) if anyone would like me to do further debugging. > > Thanks, > --- > > Greg Harris > System Administrator > MetaCarta, Inc. > > (O) +1 (617) 301-5530 > (M) +1 (781) 258-4474 > > > --- > > Greg Harris > System Administrator > MetaCarta, Inc. > > (O) +1 (617) 301-5530 > (M) +1 (781) 258-4474 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >