Re: [PATCH 0/6 v5.1] cfq-iosched: Introduce CFQ group hierarchical scheduling and "use_hierarchy" interface

[Gui Jianfeng <guijianfeng@cn.fujitsu.com>]

Gui Jianfeng wrote:
> Vivek Goyal wrote:
>> On Sun, Feb 27, 2011 at 06:16:18PM -0500, Vivek Goyal wrote:
>>> On Fri, Feb 25, 2011 at 09:55:32AM +0800, Gui Jianfeng wrote:
>>>> Vivek Goyal wrote:
>>>>> On Wed, Feb 23, 2011 at 11:01:35AM +0800, Gui Jianfeng wrote:
>>>>>> Hi
>>>>>>
>>>>>> I rebase this series on top of *for-next* branch, it will make merging life easier.
>>>>>>
>>>>>> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
>>>>>> in the way that it puts all CFQ queues in a hidden group and schedules with other 
>>>>>> CFQ group under their parent. The patchset is available here,
>>>>>> http://lkml.org/lkml/2010/8/30/30
>>>>> Gui,
>>>>>
>>>>> I was running some tests (iostest) with these patches and my system crashed
>>>>> after a while.
>>>>>
>>>>> To be precise I was running "brrmmap" test of iostest.
>>>> Vivek,
>>>>
>>>> I simply run iostest with brrmmap mode, I can't reproduce this bug.
>>>> Would you give more details.
>>>> Can you tell me the iostest command line options?
>>> iostest /dev/dm-1 -G --nrgrp 4 -m 8 --cgtime --io_serviced --dequeue --total
>>>
>>> I was actually trying to run all the workloads defined but after running
>>> 2 workloads it crashed on 3rd workload.
>>>
>>> Now I tried to re-run brrmmap and it did not crash. So I am trying to run
>>> all the inbuilt workloads again.
>>>
>>>> Did you enable use_hierarchy in root group?
>>> No I did not. Trying to test the flat setup first.
>> Again was running above job and after 3 workloads it ran into a different
>> BUG_ON().
> 
> Vivek,
> 
> It seems there's a race.
> Would you try the following patch. This patch seems working for me.
> 
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 380d667..abbbb0e 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -4126,11 +4126,12 @@ new_queue:
>  	cfqq->allocated[rw]++;
>  	cfqq->ref++;
>  
> -	spin_unlock_irqrestore(q->queue_lock, flags);
> -
>  	rq->elevator_private[0] = cic;
>  	rq->elevator_private[1] = cfqq;
>  	rq->elevator_private[2] = cfq_ref_get_cfqg(cfqq->cfqg);
> +
> +	spin_unlock_irqrestore(q->queue_lock, flags);
> +
>  	return 0;
>  
>  queue_fail:
> 
> Thanks
> Gui
> 

Jens,

This bug seems being introduced in commmit 763414b in for-next branch when
merging for-2.6.39/core branch.
Would you apply the above patch?

Vivek, can you try the patchset again with this fix? It works fine for me now.

Thanks,
Gui


> 
> 
>> Thanks
>> Vivek
>>
>> login: [277063.539001] ------------[ cut here ]------------
>> [277063.539001] kernel BUG at block/cfq-iosched.c:1407!
>> [277063.539001] invalid opcode: 0000 [#1] SMP 
>> [277063.539001] last sysfs file: /sys/devices/virtual/block/dm-1/queue/scheduler
>> [277063.539001] CPU 2 
>> [277063.539001] Modules linked in: kvm_intel kvm qla2xxx scsi_transport_fc [last unloaded: scsi_wait_scan]
>> [277063.539001] 
>> [277063.539001] Pid: 24628, comm: iostest Not tainted 2.6.38-rc4+ #3 0A98h/HP xw8600 Workstation
>> [277063.539001] RIP: 0010:[<ffffffff8121c6f0>]  [<ffffffff8121c6f0>] cfq_put_cfqg+0x13/0xc8
>> [277063.539001] RSP: 0018:ffff880129a81d48  EFLAGS: 00010046
>> [277063.539001] RAX: 0000000000000000 RBX: ffff880135e4b800 RCX: ffff88012b9d8ed0
>> [277063.539001] RDX: ffff880135e4be30 RSI: ffff880135070c00 RDI: ffff880135070c00
>> [277063.539001] RBP: ffff880129a81d58 R08: ffff880135e4bbc8 R09: ffffffff81ad76d0
>> [277063.539001] R10: ffff880129a81d48 R11: ffff880129a81d78 R12: ffff880135e4bc18
>> [277063.539001] R13: ffff880135e4bbc8 R14: ffff8801359c3020 R15: ffff880135033310
>> [277063.539001] FS:  00007f1329ed4700(0000) GS:ffff8800bfc80000(0000) knlGS:0000000000000000
>> [277063.539001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [277063.539001] CR2: 0000000000b89c08 CR3: 00000001230a5000 CR4: 00000000000006e0
>> [277063.539001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [277063.539001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [277063.539001] Process iostest (pid: 24628, threadinfo ffff880129a80000, task ffff880131394a60)
>> [277063.539001] Stack:
>> [277063.539001]  ffff880135e4b800 ffff880135e4b800 ffff880129a81d68 ffffffff8121d0a0
>> [277063.539001]  ffff880129a81db8 ffffffff8121d6b6 ffff880133af1850 ffff880135070c00
>> [277063.539001]  ffff880129a81db8 ffff88012c908900 ffff88012c908958 ffff880133af1840
>> [277063.539001] Call Trace:
>> [277063.539001]  [<ffffffff8121d0a0>] cfq_destroy_cfqg+0x45/0x47
>> [277063.539001]  [<ffffffff8121d6b6>] cfq_exit_queue+0xcc/0x164
>> [277063.539001]  [<ffffffff81209f19>] elevator_exit+0x2a/0x47
>> [277063.539001]  [<ffffffff8120a8cc>] elevator_change+0x12f/0x1b7
>> [277063.539001]  [<ffffffff8120a976>] elv_iosched_store+0x22/0x4c
>> [277063.539001]  [<ffffffff812111f2>] queue_attr_store+0x6a/0x89
>> [277063.539001]  [<ffffffff8113e093>] sysfs_write_file+0xfc/0x138
>> [277063.539001]  [<ffffffff810e9fd4>] vfs_write+0xa9/0x105
>> [277063.539001]  [<ffffffff810ea0e9>] sys_write+0x45/0x6c
>> [277063.539001]  [<ffffffff8100293b>] system_call_fastpath+0x16/0x1b
>> [277063.539001] Code: 75 09 48 85 db 0f 85 77 ff ff ff 41 5c 5b 41 5c 41 5d 41 5e 41 5f c9 c3 55 48 89 e5 53 48 83 ec 08 8b 87 20 03 00 00 85 c0 7f 04 <0f> 0b eb fe ff c8 85 c0 89 87 20 03 00 00 0f 85 9d 00 00 00 4c 
>> [277063.539001] RIP  [<ffffffff8121c6f0>] cfq_put_cfqg+0x13/0xc8
>> [277063.539001]  RSP <ffff880129a81d48>
>> [277063.539001] ---[ end trace d7596ee55221d6a7 ]---
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>