block virtio excessive VIRTIO_PCI_ISR reads

block virtio excessive VIRTIO_PCI_ISR reads

[Saul Tamari]

Hi,

There seems to be a bug in the VirtIO guest device driver (tested with
2.6.31.5 kernel).
The bug causes frequent & excessive VIRTIO_PCI_ISR reads (causing PIO
vm-exits) for a device that is not being used.

The setup I’m using includes two VirtIO block devices – vda & vdb.
While running heavy IO on vdb I notice a lot of VIRTIO_PCI_ISR port
reads for the PIO ports of vda too.
For each vdb IO operation passed to the hypervisor by the virtio driver I see:
1. One VIRTIO_PCI_QUEUE_NOTIFY write on vdb
2. One VIRTIO_PCI_ISR read on vdb
3. One VIRTIO_PCI_ISR read on vda (while not explicitly using vda)

I think that #3 shouldn't happen and I think it is a bug.


On my VM setup, vda’s IO ports start at 0xc400 and vdb's IO ports
start at 0xc800.
Below are the counts of PIO vm-exits distributed by port number:
port                count
0xc413            17923
0xc810            17922
0xc813            17923

Is this how this VirtIO block devices supposed to work? The extra ISR
reads on vda are ok?


Thanks,
Saul

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Anthony Liguori]

Saul Tamari wrote:
> Hi,
>
> There seems to be a bug in the VirtIO guest device driver (tested with
> 2.6.31.5 kernel).
> The bug causes frequent & excessive VIRTIO_PCI_ISR reads (causing PIO
> vm-exits) for a device that is not being used.
>
> The setup I’m using includes two VirtIO block devices – vda & vdb.
> While running heavy IO on vdb I notice a lot of VIRTIO_PCI_ISR port
> reads for the PIO ports of vda too.
> For each vdb IO operation passed to the hypervisor by the virtio driver I see:
> 1. One VIRTIO_PCI_QUEUE_NOTIFY write on vdb
> 2. One VIRTIO_PCI_ISR read on vdb
> 3. One VIRTIO_PCI_ISR read on vda (while not explicitly using vda)
>   

That's because it's using a shared interrupt and each device needs to 
check whether it's got pending work to do.

You can eliminate this by using MSI-x.

Regards,

Anthony Liguori

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Michael Tokarev]

Anthony Liguori wrote:
> Saul Tamari wrote:
>> Hi,
>>
>> There seems to be a bug in the VirtIO guest device driver (tested with
>> 2.6.31.5 kernel).
>> The bug causes frequent & excessive VIRTIO_PCI_ISR reads (causing PIO
>> vm-exits) for a device that is not being used.
>>
>> The setup I’m using includes two VirtIO block devices – vda & vdb.
>> While running heavy IO on vdb I notice a lot of VIRTIO_PCI_ISR port
>> reads for the PIO ports of vda too.
>> For each vdb IO operation passed to the hypervisor by the virtio 
>> driver I see:
>> 1. One VIRTIO_PCI_QUEUE_NOTIFY write on vdb
>> 2. One VIRTIO_PCI_ISR read on vdb
>> 3. One VIRTIO_PCI_ISR read on vda (while not explicitly using vda)
> 
> That's because it's using a shared interrupt and each device needs to 
> check whether it's got pending work to do.
> 
> You can eliminate this by using MSI-x.

So the next question is how to use MSI-x.. ;)

Thanks!

/mjt

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Saul Tamari]

yep!

What do I have to do to turn on MSI-x?
Make this PCI capability in QEMU's virtio??


Thanks,
Saul


On Wed, Oct 28, 2009 at 4:59 PM, Michael Tokarev <mjt@tls.msk.ru> wrote:
[snip]
>
> So the next question is how to use MSI-x.. ;)
>
> Thanks!
>
> /mjt
>

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Saul Tamari]

BTW.
The kernel I'm using has MSI-x support enabled and this same thing
also occurs on the default FC11 2.6.29.4-167 kernel.

Below is a partial dump of 'lspci -vv'.
I only pasted the output from the virtio networking device and from
the virtio block device.
You can notice that the block device doesn't have the MSI-x capability.
Is there something I can tell QEmu to add the MSI-x capability to the
virtio block device?


00:03.0 Ethernet controller: Qumranet, Inc. Virtio network device
        Subsystem: Qumranet, Inc. Device 0001
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 11
        Region 0: I/O ports at c040 [size=32]
        Region 1: Memory at f2001000 (32-bit, non-prefetchable)
[size=4K]
        Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00000800
        Kernel driver in use: virtio-pci
        Kernel modules: virtio_pci


00:05.0 Mass storage controller: Qumranet, Inc. Virtio block device
        Subsystem: Qumranet, Inc. Device 0002
        Physical Slot: 5
        Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 10
        Region 0: I/O ports at c800 [size=1K]
        Kernel driver in use: virtio-pci
        Kernel modules: virtio_pci


Thanks,
Saul



On Wed, Oct 28, 2009 at 5:04 PM, Saul Tamari <stamari@gmail.com> wrote:
> yep!
>
> What do I have to do to turn on MSI-x?
> Make this PCI capability in QEMU's virtio??
>
>
> Thanks,
> Saul
>
>
> On Wed, Oct 28, 2009 at 4:59 PM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> [snip]
>>
>> So the next question is how to use MSI-x.. ;)
>>
>> Thanks!
>>
>> /mjt
>>
>

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Avi Kivity]

On 10/28/2009 05:04 PM, Saul Tamari wrote:
> yep!
>
> What do I have to do to turn on MSI-x?
> Make this PCI capability in QEMU's virtio??
>
>    

Pick up a new qemu-kvm from git.

-- 
error compiling committee.c: too many arguments to function

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Saul Tamari]

Hi,

The current qemu-kvm from the git indeed adds the MSI-x capability to
the virtio block device.
With MSI-x enabled on the virtio block device, there aren't
VIRTIO_PCI_ISR reads on the unused device.


Thanks,
Saul


On Thu, Oct 29, 2009 at 12:10 PM, Avi Kivity <avi@redhat.com> wrote:
[snip]
>
> Pick up a new qemu-kvm from git.
>

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Avi Kivity]

On 10/29/2009 05:47 PM, Saul Tamari wrote:
> Hi,
>
> The current qemu-kvm from the git indeed adds the MSI-x capability to
> the virtio block device.
> With MSI-x enabled on the virtio block device, there aren't
> VIRTIO_PCI_ISR reads on the unused device.
>    

There shouldn't be VIRTIO_PCI_ISR reads on any device, not just the 
unused ones, if all are MSI-X capable.

-- 
error compiling committee.c: too many arguments to function

Re: block virtio excessive VIRTIO_PCI_ISR reads

[Saul Tamari]

You are right.
After enabling MSI-x for the virtio block device I now only see
VIRTIO_PCI_QUEUE_NOTIFY writes and there aren't any VIRTIO_PCI_ISR
reads anymore.


Thanks,
Saul



On Sun, Nov 1, 2009 at 11:23 AM, Avi Kivity <avi@redhat.com> wrote:
> On 10/29/2009 05:47 PM, Saul Tamari wrote:
>>
>> Hi,
>>
>> The current qemu-kvm from the git indeed adds the MSI-x capability to
>> the virtio block device.
>> With MSI-x enabled on the virtio block device, there aren't
>> VIRTIO_PCI_ISR reads on the unused device.
>>
>
> There shouldn't be VIRTIO_PCI_ISR reads on any device, not just the unused
> ones, if all are MSI-X capable.
>
> --
> error compiling committee.c: too many arguments to function
>
>