Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Jason Wang <jasowang@redhat.com>]

On 2020/8/7 下午5:59, Stefan Hajnoczi wrote:
> On Fri, Aug 07, 2020 at 11:35:08AM +0800, Jason Wang wrote:
>> On 2020/8/6 下午8:37, Stefan Hajnoczi wrote:
>>> On Wed, Aug 05, 2020 at 08:13:29AM -0400, Michael S. Tsirkin wrote:
>>>> On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
>>>>> On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
>>>>>> I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
>>>>>> Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
>>>>>> Basically increase queue size can get better RX rate for my case.
>>>>>>
>>>>>> Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
>>>>> Hi,
>>>>> The VIRTIO 1.1 specification says the maximum number of descriptors is
>>>>> 32768 for both split and packed virtqueues.
>>>>>
>>>>> The vhost kernel code seems to support 32768.
>>>>>
>>>>> The 1024 limit is an implementation limit in QEMU. Increasing it would
>>>>> require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
>>>>> the size of arrays.
>>>>>
>>>>> I can't think of a fundamental reason why QEMU needs to limit itself to
>>>>> 1024 descriptors. Raising the limit would require fixing up the code and
>>>>> ensuring that live migration remains compatible with older versions of
>>>>> QEMU.
>>>>>
>>>>> Stefan
>>>> There's actually a reason for a limit: in theory the vq size
>>>> also sets a limit on the number of scatter/gather entries.
>>>> both QEMU and vhost can't handle a packet split over > 1k chunks.
>>>>
>>>> We could add an extra limit for s/g size like block and scsi do,
>>>> this will need spec, guest and host side work.
>>> Interesting, thanks for explaining! This could be made explicit by
>>> changing the QEMU code to:
>>>
>>> include/hw/virtio/virtio.h:#define VIRTQUEUE_MAX_SIZE IOV_MAX
>>>
>>> Looking more closely at the vhost kernel code I see that UIO_MAXIOV is
>>> used in some places but not in vhost_vring_set_num() (ioctl
>>> VHOST_SET_VRING_NUM). Is there a reason why UIO_MAXIOV isn't enforced
>>> when the application sets the queue size?
>>
>> Actually three things:
>>
>> 1) queue size
>> 2) #descriptors in a list
>> 3) IOV size
>>
>> Spec limit the 2) to 1) but 2) may not equal to 3).
>>
>> So enforcing UIO_MAXIOV can not solve the problem completely.
>>
>> For vhost-net, it depends on socket to build skb which requires an iov array
>> to work. We need remove this limitation by:
>>
>> - build skb by vhost-net itself
>> - do piecewise copying
>>
>> Then we're not limited with #iov and more and support up to what spec
>> supports.
> If I understand correctly, you are saying vhost_net.ko does not support
> more than UIO_MAXIOV descriptors today but it could be fixed as you
> described.


Yes, but it needs major refactoring on vhost_net.

thanks


>
> Thanks!
>
> Stefan