* virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160!
@ 2014-10-16 15:17 Brian Foster
2014-10-17 13:27 ` Christoph Hellwig
0 siblings, 1 reply; 5+ messages in thread
From: Brian Foster @ 2014-10-16 15:17 UTC (permalink / raw)
To: linux-scsi
Hi all,
Hopefully this is the right list for this report...
I hit the following kernel bug reliably by running xfstests test
generic/234 against XFS using 10GB LVM test/scratch volumes on top of a
~100GB virtio_blk block device. The virt block device is file-backed on
the host.
>From the console:
------------[ cut here ]------------
kernel BUG at drivers/virtio/virtio_ring.c:160!
invalid opcode: 0000 [#1] SMP
Modules linked in: xfs libcrc32c cfg80211 rfkill snd_hda_codec_generic ppdev snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd serio_raw virtio_balloon soundcore virtio_console parport_pc parport i2c_piix4 sunrpc virtio_blk virtio_net qxl drm_kms_helper ttm ata_generic virtio_pci virtio_ring drm virtio pata_acpi
CPU: 0 PID: 1442 Comm: xfsaild/dm-3 Not tainted 3.17.0+ #97
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff8800c6483460 ti: ffff880118034000 task.ti: ffff880118034000
RIP: 0010:[<ffffffffa00729e5>] [<ffffffffa00729e5>] virtqueue_add_sgs+0x415/0x430 [virtio_ring]
RSP: 0018:ffff880118037678 EFLAGS: 00010002
RAX: ffff88011808b000 RBX: ffff8801180377e8 RCX: 0000000000000003
RDX: ffffea0003667102 RSI: ffff8801180377d0 RDI: ffff880118037730
RBP: ffff8801180376e8 R08: ffff8800d78caf70 R09: 0000000000000020
R10: ffff8800c6483460 R11: ffff8800c6484050 R12: ffff8801180377e8
R13: 0000000000000002 R14: 0000000000000081 R15: 0000000000000020
FS: 0000000000000000(0000) GS:ffff88011ae00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fffa85ff0c0 CR3: 0000000001e14000 CR4: 00000000000006f0
Stack:
ffff880118037698 0000000000000304 ffff8800d78caf70 ffff8801180377d0
0000000300000000 ffff88011808b000 ffff880118037730 ffff880000000002
ffff8801180376e8 ffff8800d78caf70 ffff880118037730 0000000000000002
Call Trace:
[<ffffffffa006a42f>] __virtblk_add_req+0xdf/0x1c0 [virtio_blk]
[<ffffffffa006a5f2>] ? virtio_queue_rq+0xe2/0x280 [virtio_blk]
[<ffffffffa006a616>] virtio_queue_rq+0x106/0x280 [virtio_blk]
[<ffffffff813d71f1>] __blk_mq_run_hw_queue+0x1d1/0x350
[<ffffffff813d7bb0>] blk_mq_run_hw_queue+0x70/0xa0
[<ffffffff813d8a6d>] blk_mq_insert_requests+0xfd/0x2d0
[<ffffffff813d9a2b>] blk_mq_flush_plug_list+0x13b/0x160
[<ffffffff813cd5c1>] blk_flush_plug_list+0xc1/0x240
[<ffffffff813d8f2a>] blk_sq_make_request+0x2ea/0x5d0
[<ffffffff81656395>] ? dm_get_live_table+0x5/0xb0
[<ffffffff813c7760>] generic_make_request+0xe0/0x130
[<ffffffff813c7828>] submit_bio+0x78/0x160
[<ffffffffa02ffcc6>] _xfs_buf_ioapply+0x2e6/0x420 [xfs]
[<ffffffffa0300328>] ? __xfs_buf_delwri_submit+0x1d8/0x5b0 [xfs]
[<ffffffffa02fff22>] xfs_buf_submit+0xd2/0x300 [xfs]
[<ffffffffa0300328>] __xfs_buf_delwri_submit+0x1d8/0x5b0 [xfs]
[<ffffffffa03021af>] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
[<ffffffffa03021af>] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
[<ffffffffa0340635>] xfsaild+0x275/0xe30 [xfs]
[<ffffffffa03403c0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
[<ffffffff810cda79>] kthread+0xf9/0x110
[<ffffffff810cd980>] ? kthread_create_on_node+0x250/0x250
[<ffffffff8182717c>] ret_from_fork+0x7c/0xb0
[<ffffffff810cd980>] ? kthread_create_on_node+0x250/0x250
Code: ff 0f 1f 44 00 00 eb 84 48 8b 4d b8 8b 55 b0 48 c7 c6 19 42 07 a0 48 c7 c7 78 50 07 a0 31 c0 31 db e8 10 5d 3b e1 e9 4d fd ff ff <0f> 0b bb fb ff ff ff e9 41 fd ff ff 66 66 66 66 66 66 2e 0f 1f
RIP [<ffffffffa00729e5>] virtqueue_add_sgs+0x415/0x430 [virtio_ring]
RSP <ffff880118037678>
---[ end trace 823f74f9a11abe26 ]---
This occurs on the latest tot kernel (commit 0429fbc0bdc2) but appears
to originate sometime during the 3.16 development cycle. A bisect lands
on the following commit:
05f1dd53 block: add queue flag for disabling SG merging
To corroborate that, the appended diff appears to work around the
problem for me (included as a data point, not a fix, as I'm not familiar
with the block layer). Let me know if I can provide any more info,
thanks!
Brian
---8<---
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 0a58140..5861bd72 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
vblk->tag_set.ops = &virtio_mq_ops;
vblk->tag_set.queue_depth = virtblk_queue_depth;
vblk->tag_set.numa_node = NUMA_NO_NODE;
- vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
+ vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE|BLK_MQ_F_SG_MERGE;
vblk->tag_set.cmd_size =
sizeof(struct virtblk_req) +
sizeof(struct scatterlist) * sg_elems;
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160!
2014-10-16 15:17 virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160! Brian Foster
@ 2014-10-17 13:27 ` Christoph Hellwig
2014-10-21 20:05 ` Jens Axboe
0 siblings, 1 reply; 5+ messages in thread
From: Christoph Hellwig @ 2014-10-17 13:27 UTC (permalink / raw)
To: Brian Foster, Jens Axboe; +Cc: linux-scsi
On Thu, Oct 16, 2014 at 11:17:39AM -0400, Brian Foster wrote:
> Hi all,
>
> Hopefully this is the right list for this report...
>
> I hit the following kernel bug reliably by running xfstests test
> generic/234 against XFS using 10GB LVM test/scratch volumes on top of a
> ~100GB virtio_blk block device. The virt block device is file-backed on
> the host.
Jens, I thought the segment merging bug was fixed a while ago. Did we
manage to not include parts of it for 3.17?
>
> >From the console:
>
> ------------[ cut here ]------------
> kernel BUG at drivers/virtio/virtio_ring.c:160!
> invalid opcode: 0000 [#1] SMP
> Modules linked in: xfs libcrc32c cfg80211 rfkill snd_hda_codec_generic ppdev snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd serio_raw virtio_balloon soundcore virtio_console parport_pc parport i2c_piix4 sunrpc virtio_blk virtio_net qxl drm_kms_helper ttm ata_generic virtio_pci virtio_ring drm virtio pata_acpi
> CPU: 0 PID: 1442 Comm: xfsaild/dm-3 Not tainted 3.17.0+ #97
> Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> task: ffff8800c6483460 ti: ffff880118034000 task.ti: ffff880118034000
> RIP: 0010:[<ffffffffa00729e5>] [<ffffffffa00729e5>] virtqueue_add_sgs+0x415/0x430 [virtio_ring]
> RSP: 0018:ffff880118037678 EFLAGS: 00010002
> RAX: ffff88011808b000 RBX: ffff8801180377e8 RCX: 0000000000000003
> RDX: ffffea0003667102 RSI: ffff8801180377d0 RDI: ffff880118037730
> RBP: ffff8801180376e8 R08: ffff8800d78caf70 R09: 0000000000000020
> R10: ffff8800c6483460 R11: ffff8800c6484050 R12: ffff8801180377e8
> R13: 0000000000000002 R14: 0000000000000081 R15: 0000000000000020
> FS: 0000000000000000(0000) GS:ffff88011ae00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007fffa85ff0c0 CR3: 0000000001e14000 CR4: 00000000000006f0
> Stack:
> ffff880118037698 0000000000000304 ffff8800d78caf70 ffff8801180377d0
> 0000000300000000 ffff88011808b000 ffff880118037730 ffff880000000002
> ffff8801180376e8 ffff8800d78caf70 ffff880118037730 0000000000000002
> Call Trace:
> [<ffffffffa006a42f>] __virtblk_add_req+0xdf/0x1c0 [virtio_blk]
> [<ffffffffa006a5f2>] ? virtio_queue_rq+0xe2/0x280 [virtio_blk]
> [<ffffffffa006a616>] virtio_queue_rq+0x106/0x280 [virtio_blk]
> [<ffffffff813d71f1>] __blk_mq_run_hw_queue+0x1d1/0x350
> [<ffffffff813d7bb0>] blk_mq_run_hw_queue+0x70/0xa0
> [<ffffffff813d8a6d>] blk_mq_insert_requests+0xfd/0x2d0
> [<ffffffff813d9a2b>] blk_mq_flush_plug_list+0x13b/0x160
> [<ffffffff813cd5c1>] blk_flush_plug_list+0xc1/0x240
> [<ffffffff813d8f2a>] blk_sq_make_request+0x2ea/0x5d0
> [<ffffffff81656395>] ? dm_get_live_table+0x5/0xb0
> [<ffffffff813c7760>] generic_make_request+0xe0/0x130
> [<ffffffff813c7828>] submit_bio+0x78/0x160
> [<ffffffffa02ffcc6>] _xfs_buf_ioapply+0x2e6/0x420 [xfs]
> [<ffffffffa0300328>] ? __xfs_buf_delwri_submit+0x1d8/0x5b0 [xfs]
> [<ffffffffa02fff22>] xfs_buf_submit+0xd2/0x300 [xfs]
> [<ffffffffa0300328>] __xfs_buf_delwri_submit+0x1d8/0x5b0 [xfs]
> [<ffffffffa03021af>] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
> [<ffffffffa03021af>] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
> [<ffffffffa0340635>] xfsaild+0x275/0xe30 [xfs]
> [<ffffffffa03403c0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
> [<ffffffff810cda79>] kthread+0xf9/0x110
> [<ffffffff810cd980>] ? kthread_create_on_node+0x250/0x250
> [<ffffffff8182717c>] ret_from_fork+0x7c/0xb0
> [<ffffffff810cd980>] ? kthread_create_on_node+0x250/0x250
> Code: ff 0f 1f 44 00 00 eb 84 48 8b 4d b8 8b 55 b0 48 c7 c6 19 42 07 a0 48 c7 c7 78 50 07 a0 31 c0 31 db e8 10 5d 3b e1 e9 4d fd ff ff <0f> 0b bb fb ff ff ff e9 41 fd ff ff 66 66 66 66 66 66 2e 0f 1f
> RIP [<ffffffffa00729e5>] virtqueue_add_sgs+0x415/0x430 [virtio_ring]
> RSP <ffff880118037678>
> ---[ end trace 823f74f9a11abe26 ]---
>
> This occurs on the latest tot kernel (commit 0429fbc0bdc2) but appears
> to originate sometime during the 3.16 development cycle. A bisect lands
> on the following commit:
>
> 05f1dd53 block: add queue flag for disabling SG merging
>
> To corroborate that, the appended diff appears to work around the
> problem for me (included as a data point, not a fix, as I'm not familiar
> with the block layer). Let me know if I can provide any more info,
> thanks!
>
> Brian
>
> ---8<---
>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 0a58140..5861bd72 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
> vblk->tag_set.ops = &virtio_mq_ops;
> vblk->tag_set.queue_depth = virtblk_queue_depth;
> vblk->tag_set.numa_node = NUMA_NO_NODE;
> - vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
> + vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE|BLK_MQ_F_SG_MERGE;
> vblk->tag_set.cmd_size =
> sizeof(struct virtblk_req) +
> sizeof(struct scatterlist) * sg_elems;
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
---end quoted text---
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160!
2014-10-17 13:27 ` Christoph Hellwig
@ 2014-10-21 20:05 ` Jens Axboe
2014-10-22 0:57 ` Ming Lei
0 siblings, 1 reply; 5+ messages in thread
From: Jens Axboe @ 2014-10-21 20:05 UTC (permalink / raw)
To: Christoph Hellwig, Brian Foster; +Cc: linux-scsi, Ming Lei
On 10/17/2014 07:27 AM, Christoph Hellwig wrote:
> On Thu, Oct 16, 2014 at 11:17:39AM -0400, Brian Foster wrote:
>> Hi all,
>>
>> Hopefully this is the right list for this report...
>>
>> I hit the following kernel bug reliably by running xfstests test
>> generic/234 against XFS using 10GB LVM test/scratch volumes on top of a
>> ~100GB virtio_blk block device. The virt block device is file-backed on
>> the host.
>
> Jens, I thought the segment merging bug was fixed a while ago. Did we
> manage to not include parts of it for 3.17?
Mings patch went in after 3.17, iirc. Ming?
>> ------------[ cut here ]------------
>> kernel BUG at drivers/virtio/virtio_ring.c:160!
>> invalid opcode: 0000 [#1] SMP
>> Modules linked in: xfs libcrc32c cfg80211 rfkill snd_hda_codec_generic ppdev snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd serio_raw virtio_balloon soundcore virtio_console parport_pc parport i2c_piix4 sunrpc virtio_blk virtio_net qxl drm_kms_helper ttm ata_generic virtio_pci virtio_ring drm virtio pata_acpi
>> CPU: 0 PID: 1442 Comm: xfsaild/dm-3 Not tainted 3.17.0+ #97
>> Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
>> task: ffff8800c6483460 ti: ffff880118034000 task.ti: ffff880118034000
>> RIP: 0010:[<ffffffffa00729e5>] [<ffffffffa00729e5>] virtqueue_add_sgs+0x415/0x430 [virtio_ring]
>> RSP: 0018:ffff880118037678 EFLAGS: 00010002
>> RAX: ffff88011808b000 RBX: ffff8801180377e8 RCX: 0000000000000003
>> RDX: ffffea0003667102 RSI: ffff8801180377d0 RDI: ffff880118037730
>> RBP: ffff8801180376e8 R08: ffff8800d78caf70 R09: 0000000000000020
>> R10: ffff8800c6483460 R11: ffff8800c6484050 R12: ffff8801180377e8
>> R13: 0000000000000002 R14: 0000000000000081 R15: 0000000000000020
>> FS: 0000000000000000(0000) GS:ffff88011ae00000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> CR2: 00007fffa85ff0c0 CR3: 0000000001e14000 CR4: 00000000000006f0
>> Stack:
>> ffff880118037698 0000000000000304 ffff8800d78caf70 ffff8801180377d0
>> 0000000300000000 ffff88011808b000 ffff880118037730 ffff880000000002
>> ffff8801180376e8 ffff8800d78caf70 ffff880118037730 0000000000000002
>> Call Trace:
>> [<ffffffffa006a42f>] __virtblk_add_req+0xdf/0x1c0 [virtio_blk]
>> [<ffffffffa006a5f2>] ? virtio_queue_rq+0xe2/0x280 [virtio_blk]
>> [<ffffffffa006a616>] virtio_queue_rq+0x106/0x280 [virtio_blk]
>> [<ffffffff813d71f1>] __blk_mq_run_hw_queue+0x1d1/0x350
>> [<ffffffff813d7bb0>] blk_mq_run_hw_queue+0x70/0xa0
>> [<ffffffff813d8a6d>] blk_mq_insert_requests+0xfd/0x2d0
>> [<ffffffff813d9a2b>] blk_mq_flush_plug_list+0x13b/0x160
>> [<ffffffff813cd5c1>] blk_flush_plug_list+0xc1/0x240
>> [<ffffffff813d8f2a>] blk_sq_make_request+0x2ea/0x5d0
>> [<ffffffff81656395>] ? dm_get_live_table+0x5/0xb0
>> [<ffffffff813c7760>] generic_make_request+0xe0/0x130
>> [<ffffffff813c7828>] submit_bio+0x78/0x160
>> [<ffffffffa02ffcc6>] _xfs_buf_ioapply+0x2e6/0x420 [xfs]
>> [<ffffffffa0300328>] ? __xfs_buf_delwri_submit+0x1d8/0x5b0 [xfs]
>> [<ffffffffa02fff22>] xfs_buf_submit+0xd2/0x300 [xfs]
>> [<ffffffffa0300328>] __xfs_buf_delwri_submit+0x1d8/0x5b0 [xfs]
>> [<ffffffffa03021af>] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
>> [<ffffffffa03021af>] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
>> [<ffffffffa0340635>] xfsaild+0x275/0xe30 [xfs]
>> [<ffffffffa03403c0>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
>> [<ffffffff810cda79>] kthread+0xf9/0x110
>> [<ffffffff810cd980>] ? kthread_create_on_node+0x250/0x250
>> [<ffffffff8182717c>] ret_from_fork+0x7c/0xb0
>> [<ffffffff810cd980>] ? kthread_create_on_node+0x250/0x250
>> Code: ff 0f 1f 44 00 00 eb 84 48 8b 4d b8 8b 55 b0 48 c7 c6 19 42 07 a0 48 c7 c7 78 50 07 a0 31 c0 31 db e8 10 5d 3b e1 e9 4d fd ff ff <0f> 0b bb fb ff ff ff e9 41 fd ff ff 66 66 66 66 66 66 2e 0f 1f
>> RIP [<ffffffffa00729e5>] virtqueue_add_sgs+0x415/0x430 [virtio_ring]
>> RSP <ffff880118037678>
>> ---[ end trace 823f74f9a11abe26 ]---
>>
>> This occurs on the latest tot kernel (commit 0429fbc0bdc2) but appears
>> to originate sometime during the 3.16 development cycle. A bisect lands
>> on the following commit:
>>
>> 05f1dd53 block: add queue flag for disabling SG merging
>>
>> To corroborate that, the appended diff appears to work around the
>> problem for me (included as a data point, not a fix, as I'm not familiar
>> with the block layer). Let me know if I can provide any more info,
>> thanks!
>>
>> Brian
>>
>> ---8<---
>>
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index 0a58140..5861bd72 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>> vblk->tag_set.ops = &virtio_mq_ops;
>> vblk->tag_set.queue_depth = virtblk_queue_depth;
>> vblk->tag_set.numa_node = NUMA_NO_NODE;
>> - vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
>> + vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE|BLK_MQ_F_SG_MERGE;
>> vblk->tag_set.cmd_size =
>> sizeof(struct virtblk_req) +
>> sizeof(struct scatterlist) * sg_elems;
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> ---end quoted text---
>
--
Jens Axboe
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160!
2014-10-21 20:05 ` Jens Axboe
@ 2014-10-22 0:57 ` Ming Lei
2014-10-24 17:17 ` Brian Foster
0 siblings, 1 reply; 5+ messages in thread
From: Ming Lei @ 2014-10-22 0:57 UTC (permalink / raw)
To: Jens Axboe; +Cc: Christoph Hellwig, Brian Foster, Linux SCSI List
[-- Attachment #1: Type: text/plain, Size: 822 bytes --]
On Wed, Oct 22, 2014 at 4:05 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 10/17/2014 07:27 AM, Christoph Hellwig wrote:
>> On Thu, Oct 16, 2014 at 11:17:39AM -0400, Brian Foster wrote:
>>> Hi all,
>>>
>>> Hopefully this is the right list for this report...
>>>
>>> I hit the following kernel bug reliably by running xfstests test
>>> generic/234 against XFS using 10GB LVM test/scratch volumes on top of a
>>> ~100GB virtio_blk block device. The virt block device is file-backed on
>>> the host.
>>
>> Jens, I thought the segment merging bug was fixed a while ago. Did we
>> manage to not include parts of it for 3.17?
>
> Mings patch went in after 3.17, iirc. Ming?
Sorry, that patch is wrong[1], attachment patch should fix the issue.
[1] http://marc.info/?l=linux-kernel&m=141290430004361&w=2
Thanks,
--
Ming Lei
[-- Attachment #2: 0001-blk-merge-recaculate-segment-if-it-isn-t-less-than-m.patch --]
[-- Type: text/x-patch, Size: 1560 bytes --]
From dde19549352da3fea516d89bbfa25d08b784e229 Mon Sep 17 00:00:00 2001
From: Ming Lei <tom.leiming@gmail.com>
Date: Wed, 22 Oct 2014 08:30:30 +0800
Subject: [PATCH] blk-merge: recaculate segment if it isn't less than max
segments
The problem is introduced by commit 764f612c6c3c231b(blk-merge:
don't compute bi_phys_segments from bi_vcnt for cloned bio),
and merge is needed if number of current segment isn't less than
max segments.
Strictly speaking, bio->bi_vcnt shouldn't be used here since
it may not be accurate in cases of both cloned bio or bio cloned
from, but bio_segments() is a bit expensive, and bi_vcnt is still
the biggest number, so the approach should work.
Signed-off-by: Ming Lei <tom.leiming@gmail.com>
---
block/blk-merge.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index ba99351..b3ac40a 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -99,16 +99,17 @@ void blk_recount_segments(struct request_queue *q, struct bio *bio)
{
bool no_sg_merge = !!test_bit(QUEUE_FLAG_NO_SG_MERGE,
&q->queue_flags);
+ bool merge_not_need = bio->bi_vcnt < queue_max_segments(q);
if (no_sg_merge && !bio_flagged(bio, BIO_CLONED) &&
- bio->bi_vcnt < queue_max_segments(q))
+ merge_not_need)
bio->bi_phys_segments = bio->bi_vcnt;
else {
struct bio *nxt = bio->bi_next;
bio->bi_next = NULL;
bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio,
- no_sg_merge);
+ no_sg_merge && merge_not_need);
bio->bi_next = nxt;
}
--
1.7.9.5
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160!
2014-10-22 0:57 ` Ming Lei
@ 2014-10-24 17:17 ` Brian Foster
0 siblings, 0 replies; 5+ messages in thread
From: Brian Foster @ 2014-10-24 17:17 UTC (permalink / raw)
To: Ming Lei; +Cc: Jens Axboe, Christoph Hellwig, Linux SCSI List
On Wed, Oct 22, 2014 at 08:57:07AM +0800, Ming Lei wrote:
> On Wed, Oct 22, 2014 at 4:05 AM, Jens Axboe <axboe@kernel.dk> wrote:
> > On 10/17/2014 07:27 AM, Christoph Hellwig wrote:
> >> On Thu, Oct 16, 2014 at 11:17:39AM -0400, Brian Foster wrote:
> >>> Hi all,
> >>>
> >>> Hopefully this is the right list for this report...
> >>>
> >>> I hit the following kernel bug reliably by running xfstests test
> >>> generic/234 against XFS using 10GB LVM test/scratch volumes on top of a
> >>> ~100GB virtio_blk block device. The virt block device is file-backed on
> >>> the host.
> >>
> >> Jens, I thought the segment merging bug was fixed a while ago. Did we
> >> manage to not include parts of it for 3.17?
> >
> > Mings patch went in after 3.17, iirc. Ming?
>
> Sorry, that patch is wrong[1], attachment patch should fix the issue.
>
Hi guys,
Thanks for the patch. A couple notes... I went to give this a test and
the kernel from the latest tree (commit 816fb4175c29) doesn't even boot
on my vm due to another BUG()[1].
I commented that BUG() out as an experiment and was able to proceed. As
it is, I can no longer reproduce the original problem on the latest tree
with that single modification, either via generic/234 or a few other of
the generic xfstests. I added the patch below to give that a test (still
with BUG[1] commented out) and I hit yet another bug[2] just attempting
to mount a freshly mkfs'd XFS filesystem.
Brian
[1]
------------[ cut here ]------------
kernel BUG at include/linux/virtio_config.h:125!
invalid opcode: 0000 [#1] SMP
Modules linked in: virtio_console pvpanic floppy parport snd i2c_piix4 soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc qxl drm_kms_helper virtio_blk virtio_net virtio_scsi ttm drm virtio_pci virtio_ring virtio ata_generic pata_acpi
CPU: 1 PID: 51 Comm: kworker/1:1 Not tainted 3.18.0-rc1+ #1
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Workqueue: events control_work_handler [virtio_console]
task: ffff8800daa49a40 ti: ffff880035428000 task.ti: ffff880035428000
RIP: 0010:[<ffffffffa01d8340>] [<ffffffffa01d8340>] virtio_device_ready.part.9+0x4/0x6 [virtio_console]
RSP: 0000:ffff88003542bc48 EFLAGS: 00010202
RAX: 0000000000000007 RBX: ffff8800da700e00 RCX: 0000000000006e0c
RDX: 000000000000c232 RSI: 0000000000000000 RDI: 000000000001c232
RBP: ffff88003542bc48 R08: ffffffff81fcf420 R09: 0000000000000000
R10: 0000000000000001 R11: ffff8800daa4a6e0 R12: ffff880118313600
R13: 0000000000000001 R14: ffff8800365eb000 R15: ffff8800da700e20
FS: 00007f14a4d9b880(0000) GS:ffff88011b000000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f14a2a142a8 CR3: 00000000da18d000 CR4: 00000000000006e0
Stack:
ffff88003542bcb8 ffffffffa01d78da ffff880000000001 ffffffff811039ed
ffff88003542bc88 0000000000000292 0000000000000000 000000004a7fbb3c
ffff88003542bcb8 ffff880035559a80 ffff8801183136b8 ffff880118313610
Call Trace:
[<ffffffffa01d78da>] add_port+0x40a/0x440 [virtio_console]
[<ffffffff811039ed>] ? trace_hardirqs_on+0xd/0x10
[<ffffffffa01d7cda>] control_work_handler+0x3ca/0x420 [virtio_console]
[<ffffffff810c9b1d>] process_one_work+0x20d/0x830
[<ffffffff810c9a87>] ? process_one_work+0x177/0x830
[<ffffffff810ca25b>] worker_thread+0x11b/0x490
[<ffffffff810ca140>] ? process_one_work+0x830/0x830
[<ffffffff810d03f9>] kthread+0xf9/0x110
[<ffffffff810e6286>] ? local_clock+0x16/0x30
[<ffffffff810d0300>] ? kthread_create_on_node+0x250/0x250
[<ffffffff81848cfc>] ret_from_fork+0x7c/0xb0
[<ffffffff810d0300>] ? kthread_create_on_node+0x250/0x250
Code: 89 ef e8 14 d6 ff ff 49 89 c4 4d 85 e4 0f 8f 25 ff ff ff eb b0 48 c7 c0 f4 ff ff ff e9 17 ff ff ff e8 45 11 ed e0 90 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 e8 5f e0 ff ff 48 c7 c7 c0
RIP [<ffffffffa01d8340>] virtio_device_ready.part.9+0x4/0x6 [virtio_console]
RSP <ffff88003542bc48>
---[ end trace 2b6f7b706d173ace ]---
[2]
XFS (dm-3): Mounting V4 Filesystem
------------[ cut here ]------------
kernel BUG at drivers/block/virtio_blk.c:172!
invalid opcode: 0000 [#1] SMP
Modules linked in: xfs libcrc32c xt_conntrack cfg80211 rfkill ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm ppdev joydev snd_timer serio_raw virtio_console virtio_balloon snd i2c_piix4 soundcore pvpanic parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc virtio_net virtio_blk qxl drm_kms_helper ttm virtio_pci virtio_ring virtio drm ata_generic pata_acpi
CPU: 2 PID: 1152 Comm: mount Not tainted 3.18.0-rc1+ #7
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
task: ffff8800d89d9a30 ti: ffff8800d9ffc000 task.ti: ffff8800d9ffc000
RIP: 0010:[<ffffffffa007c737>] [<ffffffffa007c737>] virtio_queue_rq+0x277/0x280 [virtio_blk]
RSP: 0018:ffff8800d9fff798 EFLAGS: 00010202
RAX: 0000000000000082 RBX: ffff8800d7152b80 RCX: dead000000200200
RDX: 0000000000000000 RSI: ffff8800d7152b80 RDI: ffff880035c05800
RBP: ffff8800d9fff7d8 R08: ffff8800d7152b80 R09: ffff8800d9fff800
R10: 0000000000000000 R11: ffff8800d89da660 R12: ffff8800d9fff800
R13: ffff880035c05800 R14: ffff8800d818c200 R15: 0000000000000000
FS: 00007f7d544a4840(0000) GS:ffff88011b200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f85265e75e0 CR3: 00000000d949e000 CR4: 00000000000006e0
Stack:
ffff880000000000 ffff8800d7152cf0 0000000000000002 ffff880035c05800
ffff8800d9fff800 0000000000000000 ffff8800d7152b80 ffff880035c05848
ffff8800d9fff848 ffffffff813de0b8 ffff880035c71940 0000000000000001
Call Trace:
[<ffffffff813de0b8>] __blk_mq_run_hw_queue+0x1c8/0x330
[<ffffffff813deb00>] blk_mq_run_hw_queue+0x70/0xa0
[<ffffffff813dfa2d>] blk_mq_insert_requests+0xfd/0x2d0
[<ffffffff813e09eb>] blk_mq_flush_plug_list+0x13b/0x160
[<ffffffff813d4451>] blk_flush_plug_list+0xc1/0x240
[<ffffffff813d4ba8>] blk_finish_plug+0x18/0x50
[<ffffffffa041e7bd>] _xfs_buf_ioapply+0x32d/0x5e0 [xfs]
[<ffffffffa04232e4>] ? xfs_bwrite+0x24/0x60 [xfs]
[<ffffffffa0422341>] xfs_buf_submit_wait+0x91/0x890 [xfs]
[<ffffffffa04232e4>] xfs_bwrite+0x24/0x60 [xfs]
[<ffffffffa0455677>] xlog_bwrite+0x87/0x110 [xfs]
[<ffffffffa0456267>] ? xlog_add_record.isra.10+0xf7/0x160 [xfs]
[<ffffffffa0456483>] xlog_write_log_records+0x1b3/0x250 [xfs]
[<ffffffffa0456628>] xlog_clear_stale_blocks+0x108/0x1b0 [xfs]
[<ffffffffa0455d33>] ? xlog_bread+0x43/0x60 [xfs]
[<ffffffffa045822b>] xlog_find_tail+0x33b/0x420 [xfs]
[<ffffffffa045d0dd>] xlog_recover+0x2d/0x130 [xfs]
[<ffffffffa044bedc>] xfs_log_mount+0x24c/0x2c0 [xfs]
[<ffffffffa04417ec>] xfs_mountfs+0x48c/0x960 [xfs]
[<ffffffffa0442d5a>] ? xfs_mru_cache_create+0x18a/0x200 [xfs]
[<ffffffffa044502a>] xfs_fs_fill_super+0x33a/0x3b0 [xfs]
[<ffffffff81267d10>] mount_bdev+0x1b0/0x1f0
[<ffffffffa0444cf0>] ? xfs_parseargs+0xbf0/0xbf0 [xfs]
[<ffffffffa04430a5>] xfs_fs_mount+0x15/0x20 [xfs]
[<ffffffff812686b9>] mount_fs+0x39/0x1b0
[<ffffffff81201435>] ? __alloc_percpu+0x15/0x20
[<ffffffff81288b3b>] vfs_kern_mount+0x6b/0x150
[<ffffffff8128bb50>] do_mount+0x230/0xbc0
[<ffffffff8120bca8>] ? might_fault+0xa8/0xb0
[<ffffffff811fb62b>] ? memdup_user+0x4b/0x90
[<ffffffff8128c832>] SyS_mount+0xa2/0x110
[<ffffffff817e0de9>] system_call_fastpath+0x12/0x17
Code: fe ff ff 90 0f b7 86 f4 00 00 00 c7 86 78 01 00 00 02 00 00 00 48 c7 86 80 01 00 00 00 00 00 00 89 86 7c 01 00 00 e9 12 fe ff ff <0f> 0b 0f 1f 80 00 00 00 00 66 66 66 66 90 55 31 f6 b9 08 00 00
RIP [<ffffffffa007c737>] virtio_queue_rq+0x277/0x280 [virtio_blk]
RSP <ffff8800d9fff798>
---[ end trace a4b47c21da10534d ]---
> [1] http://marc.info/?l=linux-kernel&m=141290430004361&w=2
>
>
> Thanks,
> --
> Ming Lei
> From dde19549352da3fea516d89bbfa25d08b784e229 Mon Sep 17 00:00:00 2001
> From: Ming Lei <tom.leiming@gmail.com>
> Date: Wed, 22 Oct 2014 08:30:30 +0800
> Subject: [PATCH] blk-merge: recaculate segment if it isn't less than max
> segments
>
> The problem is introduced by commit 764f612c6c3c231b(blk-merge:
> don't compute bi_phys_segments from bi_vcnt for cloned bio),
> and merge is needed if number of current segment isn't less than
> max segments.
>
> Strictly speaking, bio->bi_vcnt shouldn't be used here since
> it may not be accurate in cases of both cloned bio or bio cloned
> from, but bio_segments() is a bit expensive, and bi_vcnt is still
> the biggest number, so the approach should work.
>
> Signed-off-by: Ming Lei <tom.leiming@gmail.com>
> ---
> block/blk-merge.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/block/blk-merge.c b/block/blk-merge.c
> index ba99351..b3ac40a 100644
> --- a/block/blk-merge.c
> +++ b/block/blk-merge.c
> @@ -99,16 +99,17 @@ void blk_recount_segments(struct request_queue *q, struct bio *bio)
> {
> bool no_sg_merge = !!test_bit(QUEUE_FLAG_NO_SG_MERGE,
> &q->queue_flags);
> + bool merge_not_need = bio->bi_vcnt < queue_max_segments(q);
>
> if (no_sg_merge && !bio_flagged(bio, BIO_CLONED) &&
> - bio->bi_vcnt < queue_max_segments(q))
> + merge_not_need)
> bio->bi_phys_segments = bio->bi_vcnt;
> else {
> struct bio *nxt = bio->bi_next;
>
> bio->bi_next = NULL;
> bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio,
> - no_sg_merge);
> + no_sg_merge && merge_not_need);
> bio->bi_next = nxt;
> }
>
> --
> 1.7.9.5
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-10-24 17:18 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-16 15:17 virtio_blk - kernel BUG at drivers/virtio/virtio_ring.c:160! Brian Foster
2014-10-17 13:27 ` Christoph Hellwig
2014-10-21 20:05 ` Jens Axboe
2014-10-22 0:57 ` Ming Lei
2014-10-24 17:17 ` Brian Foster
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).