Jens Axboe wrote:
>>
>>http://togami.com/~warren/archive/2004/i2o_cfq_quad_bonnie.jpg
>>
>>I noticed that the call trace contained "cfq_next_request", so I was
>>curious what would happened if we changed to the deadline scheduler.
>>Booted with the same kernel but with "elevator=deadline". To our
>>surprise, bonnie++ ran simultaneously on all four I2O block devices
>>without crashing the server. For another test we tried "elevator=as"
>>and it too remained stable.
>>
>>Possible CFQ I/O scheduler problem?
> 
> 
> That looks pretty damn strange. Any chance you can capture a full oops,
> with a serial console or similar?

Fedora Core 2 x86_64 SMP (2 x Opteron) with kernel-2.6.5-1.326 (based on
2.6.6-rc1)
gcc-3.3.3-7 plus attached patch.  This panic was captured by netconsole.

http://togami.com/~warren/archive/2004/i2o_cfq_quad_bonnie.txt
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at cfq_iosched:407
invalid operand: 0000 [1] SMP
CPU 0
Pid: 1523, comm: bonnie++ Not tainted 2.6.5-1.326custom
RIP: 0010:[<ffffffff80234f93>] <ffffffff80234f93>{cfq_next_request+66}
RSP: 0000:ffffffff804a9048  EFLAGS: 00010016
RAX: 000001006cebf578 RBX: 000001007f3f1e00 RCX: 000001006ceb6340
RDX: 000001006cebf540 RSI: 0000000000000001 RDI: 000001007a225000
RBP: 000001007a225000 R08: ffffffff804a8ec8 R09: 0000000000000000
R10: 000001000289cb60 R11: 0000000000000000 R12: 000001007a225000
R13: 000001007a225000 R14: 000001001703bf58 R15: 0000000000509010
FS:  0000002a9555f360(0000) GS:ffffffff804a4e80(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000002a95558000 CR3: 0000000000101000 CR4: 00000000000006e0
Process bonnie++ (pid: 1523, stackpage=1007fd5e1c0)
Stack: 000001007aa01118 000001007d95abc0 000001007aa01118 ffffffff8022b444
         000001007d95abc0 000001007aa01118 ffffffffa0137bc0 ffffffffa0133b82
         0000000000000001 000001007d95abc0
Call Trace:<IRQ> <ffffffff8022b444>{elv_next_request+238}
<ffffffffa0133b82>{:i2o_block:i2ob_request+233}
         <ffffffffa013370e>{:i2o_block:i2o_block_reply+1162}
         <ffffffff80113d85>{handle_IRQ_event+44}
<ffffffffa00893ac>{:tg3:tg3_enable_ints+23}
         <ffffffff8013e7f9>{update_wall_time+12}
<ffffffffa002908a>{:i2o_core:i2o_run_queue+124}
         <ffffffffa002b60e>{:i2o_core:i2o_pci_interrupt+9}
<ffffffff80113d85>{handle_IRQ_event+44}
         <ffffffff801140b8>{do_IRQ+285} <ffffffff8013a894>{__do_softirq+76}
         <ffffffff8011185f>{ret_from_intr+0}  <EOI>

Code: 0f 0b 74 3a 32 80 ff ff ff ff 97 01 48 89 c8 eb 0e 48 89 de
RIP <ffffffff80234f93>{cfq_next_request+66} RSP <ffffffff804a9048>
   <0>Kernel panic: Aiee, killing interrupt handler!
In interrupt handler - not syncing
   <0>Rebooting in 5 seconds..Badness in panic at kernel/panic.c:87

Call Trace:<IRQ> <ffffffff80136024>{panic+416}
<ffffffff8011411d>{do_IRQ+386}
         <ffffffff8011185f>{ret_from_intr+0} <ffffffff80138cd3>{do_exit+87}
         <ffffffff80112ab5>{oops_end+80} <ffffffff80112bc5>{do_trap+0}
         <ffffffff80112cc2>{do_trap+253}
<ffffffff80112f45>{do_invalid_op+145}
         <ffffffff80234f93>{cfq_next_request+66}
<ffffffff8012ff68>{recalc_task_prio+343}
         <ffffffff8012ff68>{recalc_task_prio+343}
<ffffffff80111d31>{error_exit+0}
         <ffffffff80234f93>{cfq_next_request+66}
<ffffffff80156bea>{mempool_free+227}
         <ffffffff8022b444>{elv_next_request+238}
<ffffffffa0133b82>{:i2o_block:i2ob_request+233}
         <ffffffffa013370e>{:i2o_block:i2o_block_reply+1162}
         <ffffffff80113d85>{handle_IRQ_event+44}
<ffffffffa00893ac>{:tg3:tg3_enable_ints+23}
         <ffffffff8013e7f9>{update_wall_time+12}
<ffffffffa002908a>{:i2o_core:i2o_run_queue+124}
         <ffffffffa002b60e>{:i2o_core:i2o_pci_interrupt+9}
<ffffffff80113d85>{handle_IRQ_event+44}
         <ffffffff801140b8>{do_IRQ+285} <ffffffff8013a894>{__do_softirq+76}
         <ffffffff8011185f>{ret_from_intr+0}  <EOI>


> 
> A quick look at i2o_block doesn't show anything that should cause this.
> There are a number of request handling badnesses in there, though:
> 
> - kill check fo RQ_INACTIVE, it's pointless. even if it wasn't, the
>   inactive check return is buggy.
> 
> - you must not just clear req->waiting! remove that code.
> 
> - ->queue_depth should not be an expensive atomic type, you have the
>   device lock when you look at/inc/dec it.
> 
> 
>> 	/*
>>+	 *	Set up the queue
>>+	 */
>>+	for(i = 0; i < MAX_I2O_CONTROLLERS; i++)
>>+	{
>>+		i2ob_queues[i] = NULL;
>>+		i2ob_init_iop(i);
>>+	}
> 
> 
> Only initialize queues that will be used. Each allocated but unusued
> queues wastes memory.
> 

Markus implemented all of your recommendations except the atomic part,
which seemed to cause a kernel panic while loading the i2o_block driver.
   Attached is the latest i2o_block.c cleanup, while at this below URL is
his attempt of removing the atomic type.  He is not sure if he
implemented it properly.

http://i2o.shadowconnect.com/i2o_block-cleanup.patch-broke

Your recommendations would be greatly appreciated!

Thank you,
Warren Togami
wtogami@redhat.com
Markus Lidel
Markus.Lidel@shadowconnect.com