Slow boot in QEMU with virtio-scsi disks

Slow boot in QEMU with virtio-scsi disks

[Oleksandr Natalenko]

Hi.

I'd like to resurrect previous discussion [1] regarding slow kernel boot 
inside QEMU with virtio-scsi disks attached and blk_mq enabled.

Symptom:

[    2.830857] ata1: SATA max UDMA/133 abar m4096@0x98002000 port 
0x98002100 irq 36
[    2.834559] ata2: SATA max UDMA/133 abar m4096@0x98002000 port 
0x98002180 irq 36
[    2.837746] ata3: SATA max UDMA/133 abar m4096@0x98002000 port 
0x98002200 irq 36
[    2.841861] ata4: SATA max UDMA/133 abar m4096@0x98002000 port 
0x98002280 irq 36
[    2.847899] ata5: SATA max UDMA/133 abar m4096@0x98002000 port 
0x98002300 irq 36
[    2.853229] ata6: SATA max UDMA/133 abar m4096@0x98002000 port 
0x98002380 irq 36
[    3.172159] ata1: SATA link down (SStatus 0 SControl 300)
[    3.183552] ata5: SATA link down (SStatus 0 SControl 300)
[    3.189925] ata3: SATA link down (SStatus 0 SControl 300)
[    3.196156] ata6: SATA link down (SStatus 0 SControl 300)
[    3.201136] ata2: SATA link down (SStatus 0 SControl 300)
[    3.208559] ata4: SATA link down (SStatus 0 SControl 300)
[   16.480972] sd 0:0:1:0: Power-on or device reset occurred
[   16.481591] sd 0:0:0:0: [sda] 16777216 512-byte logical blocks: (8.59 
GB/8.00 GiB)
[   16.481671] sd 0:0:0:0: [sda] Write Protect is off
[   16.481815] sd 0:0:0:0: [sda] Write cache: disabled, read cache: 
enabled, doesn't support DPO or FUA
[   16.491325]  sda: sda1 sda2
[   16.517532] sd 0:0:1:0: [sdb] 16777216 512-byte logical blocks: (8.59 
GB/8.00 GiB)
[   16.525131] sr 0:0:2:0: Power-on or device reset occurred
[   16.525974] sd 0:0:1:0: [sdb] Write Protect is off
[   16.530946] sr 0:0:2:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 
cdda tray
[   16.543592] cdrom: Uniform CD-ROM driver Revision: 3.20
[   16.549815] sd 0:0:1:0: [sdb] Write cache: disabled, read cache: 
enabled, doesn't support DPO or FUA
[   16.549833] sd 0:0:0:0: [sda] Attached SCSI disk
[   16.572055]  sdb: sdb1 sdb2
[   16.580463] sd 0:0:1:0: [sdb] Attached SCSI disk

(note the hang that lasts for 13 seconds)

The disks are attached to the VM in the following manner:

-device virtio-scsi,id=scsi -device scsi-hd,drive=hd1 -drive 
if=none,media=disk,id=hd1,file=sda.img,format=raw

What I've tested so far:

* 4.14.62  + virtio-scsi +    blk_mq == slow boot
* 4.14.62  + virtio-scsi + no blk_mq == fast boot
* 4.17.13  + virtio-scsi +    blk_mq == slow boot
* 4.18-rc8 + virtio-scsi +    blk_mq == slow boot

QEMU is of v2.12.1, runs with "-machine q35,accel=kvm -cpu host". Also, 
if virtio-scsi disks are replaced with SATA disks, the hang does not 
occur (although, QEMU has other issues with SATA, but that's another 
story [3]).

Apparently, the commit that was mentioned in [2], 
b5b6e8c8d3b4cbeb447a0f10c7d5de3caa573299, forces blk_mq for virtio_scsi, 
so it cannot be disabled for new kernels.

Any hint on how to avoid this hang while still having virtio-scsi disks 
and blk_mq enabled please?

Thanks.

-- 
   Oleksandr Natalenko (post-factum)

[1] 
https://lists.gnu.org/archive/html/qemu-discuss/2018-07/msg00022.html
[2] 
https://lists.gnu.org/archive/html/qemu-discuss/2018-07/msg00037.html
[3] 
https://lists.nongnu.org/archive/html/qemu-devel/2018-05/msg06942.html

Re: Slow boot in QEMU with virtio-scsi disks

[Ming Lei]

On Sat, Aug 11, 2018 at 5:47 PM, Oleksandr Natalenko
<oleksandr@natalenko.name> wrote:
> Hi.
>
> I'd like to resurrect previous discussion [1] regarding slow kernel boot
> inside QEMU with virtio-scsi disks attached and blk_mq enabled.
>
> Symptom:
>
> [    2.830857] ata1: SATA max UDMA/133 abar m4096@0x98002000 port 0x98002100
> irq 36
> [    2.834559] ata2: SATA max UDMA/133 abar m4096@0x98002000 port 0x98002180
> irq 36
> [    2.837746] ata3: SATA max UDMA/133 abar m4096@0x98002000 port 0x98002200
> irq 36
> [    2.841861] ata4: SATA max UDMA/133 abar m4096@0x98002000 port 0x98002280
> irq 36
> [    2.847899] ata5: SATA max UDMA/133 abar m4096@0x98002000 port 0x98002300
> irq 36
> [    2.853229] ata6: SATA max UDMA/133 abar m4096@0x98002000 port 0x98002380
> irq 36
> [    3.172159] ata1: SATA link down (SStatus 0 SControl 300)
> [    3.183552] ata5: SATA link down (SStatus 0 SControl 300)
> [    3.189925] ata3: SATA link down (SStatus 0 SControl 300)
> [    3.196156] ata6: SATA link down (SStatus 0 SControl 300)
> [    3.201136] ata2: SATA link down (SStatus 0 SControl 300)
> [    3.208559] ata4: SATA link down (SStatus 0 SControl 300)
> [   16.480972] sd 0:0:1:0: Power-on or device reset occurred
> [   16.481591] sd 0:0:0:0: [sda] 16777216 512-byte logical blocks: (8.59
> GB/8.00 GiB)
> [   16.481671] sd 0:0:0:0: [sda] Write Protect is off
> [   16.481815] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled,
> doesn't support DPO or FUA
> [   16.491325]  sda: sda1 sda2
> [   16.517532] sd 0:0:1:0: [sdb] 16777216 512-byte logical blocks: (8.59
> GB/8.00 GiB)
> [   16.525131] sr 0:0:2:0: Power-on or device reset occurred
> [   16.525974] sd 0:0:1:0: [sdb] Write Protect is off
> [   16.530946] sr 0:0:2:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2
> cdda tray
> [   16.543592] cdrom: Uniform CD-ROM driver Revision: 3.20
> [   16.549815] sd 0:0:1:0: [sdb] Write cache: disabled, read cache: enabled,
> doesn't support DPO or FUA
> [   16.549833] sd 0:0:0:0: [sda] Attached SCSI disk
> [   16.572055]  sdb: sdb1 sdb2
> [   16.580463] sd 0:0:1:0: [sdb] Attached SCSI disk
>
> (note the hang that lasts for 13 seconds)
>
> The disks are attached to the VM in the following manner:
>
> -device virtio-scsi,id=scsi -device scsi-hd,drive=hd1 -drive
> if=none,media=disk,id=hd1,file=sda.img,format=raw
>
> What I've tested so far:
>
> * 4.14.62  + virtio-scsi +    blk_mq == slow boot
> * 4.14.62  + virtio-scsi + no blk_mq == fast boot
> * 4.17.13  + virtio-scsi +    blk_mq == slow boot
> * 4.18-rc8 + virtio-scsi +    blk_mq == slow boot
>
> QEMU is of v2.12.1, runs with "-machine q35,accel=kvm -cpu host". Also, if
> virtio-scsi disks are replaced with SATA disks, the hang does not occur
> (although, QEMU has other issues with SATA, but that's another story [3]).
>
> Apparently, the commit that was mentioned in [2],
> b5b6e8c8d3b4cbeb447a0f10c7d5de3caa573299, forces blk_mq for virtio_scsi, so
> it cannot be disabled for new kernels.
>
> Any hint on how to avoid this hang while still having virtio-scsi disks and
> blk_mq enabled please?

Please test for-4.19/block:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-4.19/block

This slow boot issue should have been fixed by the following commits:

1311326cf475 blk-mq: avoid to synchronize rcu inside blk_cleanup_queue()
97889f9ac24f blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()
5815839b3ca1 blk-mq: introduce new lock for protecting hctx->dispatch_wait
2278d69f030f blk-mq: don't pass **hctx to blk_mq_mark_tag_wait()
8ab6bb9ee8d0 blk-mq: cleanup blk_mq_get_driver_tag()


Thanks,
Ming Lei

Re: Slow boot in QEMU with virtio-scsi disks

[Oleksandr Natalenko]

Hi.

On 11.08.2018 14:23, Ming Lei wrote:
> Please test for-4.19/block:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-4.19/block
> 
> This slow boot issue should have been fixed by the following commits:
> 
> 1311326cf475 blk-mq: avoid to synchronize rcu inside 
> blk_cleanup_queue()
> 97889f9ac24f blk-mq: remove synchronize_rcu() from 
> blk_mq_del_queue_tag_set()
> 5815839b3ca1 blk-mq: introduce new lock for protecting 
> hctx->dispatch_wait
> 2278d69f030f blk-mq: don't pass **hctx to blk_mq_mark_tag_wait()
> 8ab6bb9ee8d0 blk-mq: cleanup blk_mq_get_driver_tag()

Indeed, I can confirm that these commits fix the issue.

Thanks a lot.

-- 
   Oleksandr Natalenko (post-factum)

Re: Slow boot in QEMU with virtio-scsi disks

[Greg Kurz]

On Sat, 11 Aug 2018 19:39:56 +0200
Oleksandr Natalenko <oleksandr@natalenko.name> wrote:

> Hi.
> 
> On 11.08.2018 14:23, Ming Lei wrote:
> > Please test for-4.19/block:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-4.19/block
> > 
> > This slow boot issue should have been fixed by the following commits:
> > 
> > 1311326cf475 blk-mq: avoid to synchronize rcu inside 
> > blk_cleanup_queue()
> > 97889f9ac24f blk-mq: remove synchronize_rcu() from 
> > blk_mq_del_queue_tag_set()
> > 5815839b3ca1 blk-mq: introduce new lock for protecting 
> > hctx->dispatch_wait
> > 2278d69f030f blk-mq: don't pass **hctx to blk_mq_mark_tag_wait()
> > 8ab6bb9ee8d0 blk-mq: cleanup blk_mq_get_driver_tag()  
> 
> Indeed, I can confirm that these commits fix the issue.
> 
> Thanks a lot.
> 

So do I.

Thanks Ming Lei for the fix !

Cheers,

--
Greg