virtio-blk-pci chokes on some PCI slots

virtio-blk-pci chokes on some PCI slots

[Markus Armbruster]

I'm working on make PCI slots configurable in QEMU.  While testing the
feature for virtio-blk-pci, I ran into a task hang with Fedora kernel
2.6.27.5-117.fc10.  Marcelo tested it with a newer kernel, and will
follow up with details.

Instead of the full QEMU patch I'm working on, the appended little hack
suffices to reproduce the problem.

    $ QEMU_VIRTIO_BLK=9 qemu foo.qcow2 -drive file=bar.img,if=virtio -serial stdio
[...]
    Loading virtio_pci module
    ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
    virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
    virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
    virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
    Loading virtio_blk module
     vda:
[tick-tock-tick-tock...]
    <3>INFO: task modprobe:491 blocked for more than 120 seconds.
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    modprobe      D c04303f1     0   491      1
           c6baeae8 00000082 c6baea80 c04303f1 c6bae000 c087667c c0879c00 c0879c00 
           c0879c00 c6b659b0 c6b65c24 c1108c00 00000000 c1108c00 83a43d86 00000002 
           c781aaa8 c6baead0 c04418d0 c6b65c24 00004d88 c781aaa8 00004d88 c6baeb3c 
    Call Trace:
     [<c04303f1>] ? irq_exit+0x5f/0x83
     [<c04418d0>] ? getnstimeofday+0x3c/0xc9
     [<c043fcae>] ? ktime_get_ts+0x4a/0x4e
     [<c06a6110>] io_schedule+0x52/0x8a
     [<c046da7d>] sync_page+0x46/0x4c
     [<c06a64c9>] __wait_on_bit_lock+0x34/0x5e
     [<c046da37>] ? sync_page+0x0/0x4c
     [<c046d9f3>] __lock_page+0x78/0x81
     [<c043cd89>] ? wake_bit_function+0x0/0x43
     [<c046da28>] lock_page+0x2c/0x2f
     [<c046dff4>] read_cache_page_async+0xa4/0xfb
     [<c04b1cec>] ? blkdev_readpage+0x0/0x11
     [<c046e057>] read_cache_page+0xc/0x3c
     [<c04c8ec1>] read_dev_sector+0x34/0x6a
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04ca945>] read_lba+0x69/0xc6
     [<c04cabde>] ? efi_partition+0x84/0x676
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04cabfd>] efi_partition+0xa3/0x676
     [<c042c71a>] ? vprintk+0x2c8/0x2f3
     [<c04a13ab>] ? iget5_locked+0x33/0x114
     [<c051cf05>] ? snprintf+0x15/0x17
     [<c04cab5a>] ? efi_partition+0x0/0x676
     [<c04c94db>] rescan_partitions+0x106/0x242
     [<c041d8ba>] ? need_resched+0x18/0x22
     [<c06a6192>] ? _cond_resched+0x8/0x32
     [<c04b1a4a>] do_open+0x1ff/0x290
     [<c04b1b9d>] __blkdev_get+0x71/0x7c
     [<c04b1bb5>] blkdev_get+0xd/0xf
     [<c04c8fe7>] register_disk+0xf0/0x141
     [<c05111cf>] add_disk+0x34/0x89
     [<c05101fc>] ? exact_match+0x0/0xb
     [<c0510e13>] ? exact_lock+0x0/0x11
     [<c88223b2>] virtblk_probe+0x275/0x2a8 [virtio_blk]
     [<c062236b>] virtio_dev_probe+0x89/0xb3
     [<c05937d0>] driver_probe_device+0xa0/0x136
     [<c05938a0>] __driver_attach+0x3a/0x59
     [<c05931f6>] bus_for_each_dev+0x3b/0x63
     [<c0593675>] driver_attach+0x14/0x16
     [<c0593866>] ? __driver_attach+0x0/0x59
     [<c0592c6a>] bus_add_driver+0x9d/0x1ba
     [<c0593a27>] driver_register+0x81/0xe1
     [<c05112ec>] ? register_blkdev+0xc8/0xdc
     [<c062245a>] register_virtio_driver+0x1f/0x21
     [<c8826022>] init+0x22/0x24 [virtio_blk]
     [<c0401125>] _stext+0x3d/0x115
     [<c8826000>] ? init+0x0/0x24 [virtio_blk]
     [<c044bf7d>] sys_init_module+0x87/0x178
     [<c0403c76>] syscall_call+0x7/0xb
     =======================

Some slots work, e.g. 5, some don't, e.g. 6.

What's going on here?


diff --git a/hw/pc.c b/hw/pc.c
index 143b697..15c44ac 100644
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -1146,7 +1146,8 @@ static void pc_init1(ram_addr_t ram_size,
         int unit_id = 0;
 
         while ((index = drive_get_index(IF_VIRTIO, 0, unit_id)) != -1) {
-            pci_create_simple(pci_bus, -1, "virtio-blk-pci");
+            char *devfn = getenv("QEMU_VIRTIO_BLK");
+            pci_create_simple(pci_bus, devfn ? atoi(devfn) * 8: -1, "virtio-blk-pci");
             unit_id++;
         }
     }

Re: virtio-blk-pci chokes on some PCI slots

[Marcelo Tosatti]

On Tue, Jun 16, 2009 at 06:47:45PM +0200, Markus Armbruster wrote:
> I'm working on make PCI slots configurable in QEMU.  While testing the
> feature for virtio-blk-pci, I ran into a task hang with Fedora kernel
> 2.6.27.5-117.fc10.  Marcelo tested it with a newer kernel, and will
> follow up with details.
> 
> Instead of the full QEMU patch I'm working on, the appended little hack
> suffices to reproduce the problem.
> 
>     $ QEMU_VIRTIO_BLK=9 qemu foo.qcow2 -drive file=bar.img,if=virtio -serial stdio
> [...]
>     Loading virtio_pci module
>     ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
>     virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
>     virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
>     virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
>     Loading virtio_blk module
>      vda:
> [tick-tock-tick-tock...]
>     <3>INFO: task modprobe:491 blocked for more than 120 seconds.
>     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>     modprobe      D c04303f1     0   491      1
>            c6baeae8 00000082 c6baea80 c04303f1 c6bae000 c087667c c0879c00 c0879c00 
>            c0879c00 c6b659b0 c6b65c24 c1108c00 00000000 c1108c00 83a43d86 00000002 
>            c781aaa8 c6baead0 c04418d0 c6b65c24 00004d88 c781aaa8 00004d88 c6baeb3c 
>     Call Trace:
>      [<c04303f1>] ? irq_exit+0x5f/0x83
>      [<c04418d0>] ? getnstimeofday+0x3c/0xc9
>      [<c043fcae>] ? ktime_get_ts+0x4a/0x4e
>      [<c06a6110>] io_schedule+0x52/0x8a
>      [<c046da7d>] sync_page+0x46/0x4c
>      [<c06a64c9>] __wait_on_bit_lock+0x34/0x5e
>      [<c046da37>] ? sync_page+0x0/0x4c
>      [<c046d9f3>] __lock_page+0x78/0x81
>      [<c043cd89>] ? wake_bit_function+0x0/0x43
>      [<c046da28>] lock_page+0x2c/0x2f
>      [<c046dff4>] read_cache_page_async+0xa4/0xfb
>      [<c04b1cec>] ? blkdev_readpage+0x0/0x11
>      [<c046e057>] read_cache_page+0xc/0x3c
>      [<c04c8ec1>] read_dev_sector+0x34/0x6a
>      [<c04cab5a>] ? efi_partition+0x0/0x676
>      [<c04ca945>] read_lba+0x69/0xc6
>      [<c04cabde>] ? efi_partition+0x84/0x676
>      [<c04cab5a>] ? efi_partition+0x0/0x676
>      [<c04cabfd>] efi_partition+0xa3/0x676
>      [<c042c71a>] ? vprintk+0x2c8/0x2f3
>      [<c04a13ab>] ? iget5_locked+0x33/0x114
>      [<c051cf05>] ? snprintf+0x15/0x17
>      [<c04cab5a>] ? efi_partition+0x0/0x676
>      [<c04c94db>] rescan_partitions+0x106/0x242
>      [<c041d8ba>] ? need_resched+0x18/0x22
>      [<c06a6192>] ? _cond_resched+0x8/0x32
>      [<c04b1a4a>] do_open+0x1ff/0x290
>      [<c04b1b9d>] __blkdev_get+0x71/0x7c
>      [<c04b1bb5>] blkdev_get+0xd/0xf
>      [<c04c8fe7>] register_disk+0xf0/0x141
>      [<c05111cf>] add_disk+0x34/0x89
>      [<c05101fc>] ? exact_match+0x0/0xb
>      [<c0510e13>] ? exact_lock+0x0/0x11
>      [<c88223b2>] virtblk_probe+0x275/0x2a8 [virtio_blk]
>      [<c062236b>] virtio_dev_probe+0x89/0xb3
>      [<c05937d0>] driver_probe_device+0xa0/0x136
>      [<c05938a0>] __driver_attach+0x3a/0x59
>      [<c05931f6>] bus_for_each_dev+0x3b/0x63
>      [<c0593675>] driver_attach+0x14/0x16
>      [<c0593866>] ? __driver_attach+0x0/0x59
>      [<c0592c6a>] bus_add_driver+0x9d/0x1ba
>      [<c0593a27>] driver_register+0x81/0xe1
>      [<c05112ec>] ? register_blkdev+0xc8/0xdc
>      [<c062245a>] register_virtio_driver+0x1f/0x21
>      [<c8826022>] init+0x22/0x24 [virtio_blk]
>      [<c0401125>] _stext+0x3d/0x115
>      [<c8826000>] ? init+0x0/0x24 [virtio_blk]
>      [<c044bf7d>] sys_init_module+0x87/0x178
>      [<c0403c76>] syscall_call+0x7/0xb
>      =======================
> 
> Some slots work, e.g. 5, some don't, e.g. 6.
> 
> What's going on here?
> 
> 
> diff --git a/hw/pc.c b/hw/pc.c
> index 143b697..15c44ac 100644
> --- a/hw/pc.c
> +++ b/hw/pc.c
> @@ -1146,7 +1146,8 @@ static void pc_init1(ram_addr_t ram_size,
>          int unit_id = 0;
>  
>          while ((index = drive_get_index(IF_VIRTIO, 0, unit_id)) != -1) {
> -            pci_create_simple(pci_bus, -1, "virtio-blk-pci");
> +            char *devfn = getenv("QEMU_VIRTIO_BLK");
> +            pci_create_simple(pci_bus, devfn ? atoi(devfn) * 8: -1, "virtio-blk-pci");
>              unit_id++;
>          }
>      }

no softlockup with 2.6.29-06638-g17db0a0, but not very friendly either
(note the backing device has no partitions in my case, perhaps that
makes a difference).

virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low)
-> IRQ 10
virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11

Starting udev:  Wait timeout. Will continue in the background.[FAILED]

[root@guest ~]# dmesg | grep vda
 vda:<6>EXT3 FS on dm-0, internal journal

[root@guest ~]# mount /dev/vda /mnt

Hangs there forever.

Re: virtio-blk-pci chokes on some PCI slots

[Avi Kivity]

On 06/16/2009 07:47 PM, Markus Armbruster wrote:
> I'm working on make PCI slots configurable in QEMU.  While testing the
> feature for virtio-blk-pci, I ran into a task hang with Fedora kernel
> 2.6.27.5-117.fc10.  Marcelo tested it with a newer kernel, and will
> follow up with details.
>
>
> Some slots work, e.g. 5, some don't, e.g. 6.
>
> What's going on here?
>
>    

Most likely the interrupt wiring is screwed up.  Check your connectors, 
clear the motherboard of dust (dust + moisture can short out 
conductors).  If that doesn't work, check if the ACPI interrupt routing 
matches the actual routing by the pci code.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: virtio-blk-pci chokes on some PCI slots

[Markus Armbruster]

Marcelo Tosatti <mtosatti@redhat.com> writes:

> On Tue, Jun 16, 2009 at 06:47:45PM +0200, Markus Armbruster wrote:
>> I'm working on make PCI slots configurable in QEMU.  While testing the
>> feature for virtio-blk-pci, I ran into a task hang with Fedora kernel
>> 2.6.27.5-117.fc10.  Marcelo tested it with a newer kernel, and will
>> follow up with details.
>> 
>> Instead of the full QEMU patch I'm working on, the appended little hack
>> suffices to reproduce the problem.
>> 
>>     $ QEMU_VIRTIO_BLK=9 qemu foo.qcow2 -drive file=bar.img,if=virtio -serial stdio
>> [...]
>>     Loading virtio_pci module
>>     ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
>>     virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, low) -> IRQ 11
>>     virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
>>     virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
>>     Loading virtio_blk module
>>      vda:
>> [tick-tock-tick-tock...]
>>     <3>INFO: task modprobe:491 blocked for more than 120 seconds.
>>     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>     modprobe      D c04303f1     0   491      1
>>            c6baeae8 00000082 c6baea80 c04303f1 c6bae000 c087667c c0879c00 c0879c00 
>>            c0879c00 c6b659b0 c6b65c24 c1108c00 00000000 c1108c00 83a43d86 00000002 
>>            c781aaa8 c6baead0 c04418d0 c6b65c24 00004d88 c781aaa8 00004d88 c6baeb3c 
>>     Call Trace:
>>      [<c04303f1>] ? irq_exit+0x5f/0x83
>>      [<c04418d0>] ? getnstimeofday+0x3c/0xc9
>>      [<c043fcae>] ? ktime_get_ts+0x4a/0x4e
>>      [<c06a6110>] io_schedule+0x52/0x8a
>>      [<c046da7d>] sync_page+0x46/0x4c
>>      [<c06a64c9>] __wait_on_bit_lock+0x34/0x5e
>>      [<c046da37>] ? sync_page+0x0/0x4c
>>      [<c046d9f3>] __lock_page+0x78/0x81
>>      [<c043cd89>] ? wake_bit_function+0x0/0x43
>>      [<c046da28>] lock_page+0x2c/0x2f
>>      [<c046dff4>] read_cache_page_async+0xa4/0xfb
>>      [<c04b1cec>] ? blkdev_readpage+0x0/0x11
>>      [<c046e057>] read_cache_page+0xc/0x3c
>>      [<c04c8ec1>] read_dev_sector+0x34/0x6a
>>      [<c04cab5a>] ? efi_partition+0x0/0x676
>>      [<c04ca945>] read_lba+0x69/0xc6
>>      [<c04cabde>] ? efi_partition+0x84/0x676
>>      [<c04cab5a>] ? efi_partition+0x0/0x676
>>      [<c04cabfd>] efi_partition+0xa3/0x676
>>      [<c042c71a>] ? vprintk+0x2c8/0x2f3
>>      [<c04a13ab>] ? iget5_locked+0x33/0x114
>>      [<c051cf05>] ? snprintf+0x15/0x17
>>      [<c04cab5a>] ? efi_partition+0x0/0x676
>>      [<c04c94db>] rescan_partitions+0x106/0x242
>>      [<c041d8ba>] ? need_resched+0x18/0x22
>>      [<c06a6192>] ? _cond_resched+0x8/0x32
>>      [<c04b1a4a>] do_open+0x1ff/0x290
>>      [<c04b1b9d>] __blkdev_get+0x71/0x7c
>>      [<c04b1bb5>] blkdev_get+0xd/0xf
>>      [<c04c8fe7>] register_disk+0xf0/0x141
>>      [<c05111cf>] add_disk+0x34/0x89
>>      [<c05101fc>] ? exact_match+0x0/0xb
>>      [<c0510e13>] ? exact_lock+0x0/0x11
>>      [<c88223b2>] virtblk_probe+0x275/0x2a8 [virtio_blk]
>>      [<c062236b>] virtio_dev_probe+0x89/0xb3
>>      [<c05937d0>] driver_probe_device+0xa0/0x136
>>      [<c05938a0>] __driver_attach+0x3a/0x59
>>      [<c05931f6>] bus_for_each_dev+0x3b/0x63
>>      [<c0593675>] driver_attach+0x14/0x16
>>      [<c0593866>] ? __driver_attach+0x0/0x59
>>      [<c0592c6a>] bus_add_driver+0x9d/0x1ba
>>      [<c0593a27>] driver_register+0x81/0xe1
>>      [<c05112ec>] ? register_blkdev+0xc8/0xdc
>>      [<c062245a>] register_virtio_driver+0x1f/0x21
>>      [<c8826022>] init+0x22/0x24 [virtio_blk]
>>      [<c0401125>] _stext+0x3d/0x115
>>      [<c8826000>] ? init+0x0/0x24 [virtio_blk]
>>      [<c044bf7d>] sys_init_module+0x87/0x178
>>      [<c0403c76>] syscall_call+0x7/0xb
>>      =======================
>> 
>> Some slots work, e.g. 5, some don't, e.g. 6.
>> 
>> What's going on here?
>> 
>> 
>> diff --git a/hw/pc.c b/hw/pc.c
>> index 143b697..15c44ac 100644
>> --- a/hw/pc.c
>> +++ b/hw/pc.c
>> @@ -1146,7 +1146,8 @@ static void pc_init1(ram_addr_t ram_size,
>>          int unit_id = 0;
>>  
>>          while ((index = drive_get_index(IF_VIRTIO, 0, unit_id)) != -1) {
>> -            pci_create_simple(pci_bus, -1, "virtio-blk-pci");
>> +            char *devfn = getenv("QEMU_VIRTIO_BLK");
>> +            pci_create_simple(pci_bus, devfn ? atoi(devfn) * 8: -1, "virtio-blk-pci");
>>              unit_id++;
>>          }
>>      }
>
> no softlockup with 2.6.29-06638-g17db0a0, but not very friendly either
> (note the backing device has no partitions in my case, perhaps that
> makes a difference).
>
> virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low)
> -> IRQ 10
> virtio-pci 0000:00:09.0: can't derive routing for PCI INT A
> virtio-pci 0000:00:09.0: PCI INT A: no GSI - using IRQ 11
>
> Starting udev:  Wait timeout. Will continue in the background.[FAILED]
>
> [root@guest ~]# dmesg | grep vda
>  vda:<6>EXT3 FS on dm-0, internal journal
>
> [root@guest ~]# mount /dev/vda /mnt
>
> Hangs there forever.

Even better: I can reproduce it with an unpatched upstream QEMU
(qemu.git commit cfde4bd9) like this:

    $ qemu foo.qcow2 -net nic -net nic -net nic -drive file=bar.img,if=virtio

virtio-blk-pci ends up in slot 6.

Re: virtio-blk-pci chokes on some PCI slots

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

> Even better: I can reproduce it with an unpatched upstream QEMU
> (qemu.git commit cfde4bd9) like this:
>
>     $ qemu foo.qcow2 -net nic -net nic -net nic -drive file=bar.img,if=virtio
>
> virtio-blk-pci ends up in slot 6.

And:

    $ qemu foo.qcow2 -net nic,model=virtio -net nic,model=virtio -net nic,model=virtio -net nic,model=virtio

chokes like this:

    Setting up hotplug.
    Creating block device nodes.
    Creating character device nodes.
    Loading virtio_pci module
    ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
    virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, low) -> IRQ 11
    ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
    virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low) -> IRQ 10
    ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
    virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, low) -> IRQ 11
    virtio-pci 0000:00:06.0: can't derive routing for PCI INT A
    virtio-pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9
    virtio-pci 0000:00:07.0: can't derive routing for PCI INT A
    virtio-pci 0000:00:07.0: PCI INT A: no GSI - using IRQ 11
    Loading virtio_blk module

Boot completes, but none of the four NICs work (no /dev/eth*).  If I use
only two, they end up in "good" slots, and both work fine.