Re: [Qemu-devel] [PATCH V4 00/19] Support more virtio queues

[Jason Wang <jasowang@redhat.com>]

On Wed, Mar 18, 2015 at 8:58 PM, Michael S. Tsirkin <mst@redhat.com> 
wrote:
> On Wed, Mar 18, 2015 at 05:34:50PM +0800, Jason Wang wrote:
>>  We current limit the max virtio queues to 64. This is not sufficient
>>  to support multiqueue devices (e.g recent Linux support up to 256
>>  tap queues). So this series tries to let virtio to support more 
>> queues.
>>  
>>  No much works need to be done except:
>>  
>>  - Introducing transport specific queue limitation.
>>  - Let each virtio transport to use specific limit.
>>  - Speedup the MSI-X masking and unmasking through per vector queue
>>    list, and increase the maximum MSI-X vectors supported by qemu.
>>  - With the above enhancements, increase the maximum number of
>>    virtqueues supported by PCI from 64 to 513.
>>  - Compat the changes for legacy machine types.
> 
> What are the compatibility considerations here?

Two considerations:
1) To keep msix bar size to 4K for legacy machine types
2) Limit the pci queue max to 64 for legacy machine types

> 
> 
>>  With this patch, we can support up to 256 queues. Since x86 can only
>>  allow about 240 interrupt vectors for MSI-X, current Linux driver 
>> can
>>  only have about 80 queue pairs has their private MSI-X interrupt
>>  vectors. With sharing IRQ with queue pairs (RFC posted in
>>  https://lkml.org/lkml/2014/12/25/169), Linux driver can have up
>>  to about 186 queue pairs has their private MSI-X interrupt vectors.
>>  
>>  Stress/migration test on virtio-pci, compile test on other
>>  targets. And make check on s390x-softmmu and ppc64-softmmu.
>>  
>>  Cc: Paolo Bonzini <pbonzini@redhat.com>
>>  Cc: Richard Henderson <rth@twiddle.net>
>>  Cc: Michael S. Tsirkin <mst@redhat.com>
>>  Cc: Alexander Graf <agraf@suse.de>
>>  Cc: Keith Busch <keith.busch@intel.com>
>>  Cc: Kevin Wolf <kwolf@redhat.com>
>>  Cc: Stefan Hajnoczi <stefanha@redhat.com>
>>  Cc: Christian Borntraeger <borntraeger@de.ibm.com>
>>  Cc: Cornelia Huck <cornelia.huck@de.ibm.com>
>>  Cc: Amit Shah <amit.shah@redhat.com>
>>  Cc: qemu-ppc@nongnu.org
>>  
>>  Please review
>>  
>>  Thanks
>>  
>>  Changes from V3:
>>  - rebase to master and target to 2.4
>>  - handling compat issues for spapr
>>  - fixes for hmp command completion when we have a nic with 256 
>> queues
>>  - using VIRTIO_NO_VECTOR instead of 0 for invalid virtqueue in
>>    virtio-ccw
>>  - fix compile issues for ppc64-softmmu
>>  - don't export VIRTIO_CCW_QUEUE_MAX by introducing a gsi limit and
>>    inherit in ccw
>>  - use transport specific queue limit in virtio-serial
>>  - correct the stale comment for AdapterRoutes and move it to ccw 
>> patch
>>  - replace 128 with queue_max + 64 and add a comment for this in
>>    virtio_ccw_notify()
>>  
>>  Changes from V2:
>>  - move transport specific limitation to their implementation. (The
>>    left is VIRTIO_CCW_QUEUE_MAX since I fail to find a common header
>>    files other than virtio.h)
>>  - use virtio_get_queue_max() if possible in virtio.c
>>  - AdapterRoutes should use ccw limit
>>  - introduce a vector to queue mapping for virito devices and speedup
>>    the MSI-X masking and unmasking through this.
>>  
>>  Changes from V1:
>>  - add a validation against the bus limitation
>>  - switch to use a bus specific queue limit instead of a global one,
>>    this will allow us to just increase the limit of one transport
>>    without disturbing others.
>>  - only increase the queue limit of virtio-pci
>>  - limit the maximum number of virtio queues to 64 for legacy machine
>>    types
>>  
>>  Jason Wang (19):
>>    pc: add 2.4 machine types
>>    spapr: add machine type specific instance init function
>>    ppc: spapr: add 2.4 machine type
>>    monitor: replace the magic number 255 with MAX_QUEUE_NUM
>>    monitor: check return value of qemu_find_net_clients_except()
>>    virtio-ccw: using VIRTIO_NO_VECTOR instead of 0 for invalid 
>> virtqueue
>>    virtio-net: validate backend queue numbers against bus limitation
>>    virtio-net: fix the upper bound when trying to delete queues
>>    virito: introduce bus specific queue limit
>>    virtio-ccw: introduce ccw specific queue limit
>>    virtio-s390: switch to bus specific queue limit
>>    virtio-mmio: switch to bus specific queue limit
>>    virtio-pci: switch to use bus specific queue limit
>>    virtio: introduce vector to virtqueues mapping
>>    virtio: introduce virtio_queue_get_index()
>>    virtio-pci: speedup MSI-X masking and unmasking
>>    virtio-pci: increase the maximum number of virtqueues to 513
>>    pci: remove hard-coded bar size in msix_init_exclusive_bar()
>>    virtio-pci: introduce auto_msix_bar_size property
>>  
>>   hw/block/nvme.c                |  2 +-
>>   hw/char/virtio-serial-bus.c    |  2 +-
>>   hw/i386/pc_piix.c              | 42 ++++++++++++++++++++---
>>   hw/i386/pc_q35.c               | 39 +++++++++++++++++++--
>>   hw/misc/ivshmem.c              |  2 +-
>>   hw/net/virtio-net.c            |  9 ++++-
>>   hw/pci/msix.c                  | 18 ++++------
>>   hw/ppc/spapr.c                 | 70 
>> +++++++++++++++++++++++++++++++++++--
>>   hw/s390x/s390-virtio-bus.c     |  7 ++--
>>   hw/s390x/s390-virtio-ccw.c     |  7 ++--
>>   hw/s390x/virtio-ccw.c          | 20 +++++++----
>>   hw/scsi/virtio-scsi.c          |  4 +--
>>   hw/virtio/virtio-mmio.c        |  7 ++--
>>   hw/virtio/virtio-pci.c         | 78 
>> ++++++++++++++++++++++++++++--------------
>>   hw/virtio/virtio-pci.h         |  3 ++
>>   hw/virtio/virtio.c             | 75 
>> +++++++++++++++++++++++++++++++---------
>>   include/hw/compat.h            | 11 ++++++
>>   include/hw/pci/msix.h          |  2 +-
>>   include/hw/s390x/s390_flic.h   |  4 ++-
>>   include/hw/virtio/virtio-bus.h |  2 ++
>>   include/hw/virtio/virtio.h     |  7 ++--
>>   monitor.c                      | 25 ++++++++------
>>   22 files changed, 339 insertions(+), 97 deletions(-)
>>  
>>  -- 
>>  2.1.0