Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Yajun Wu]

[-- Attachment #1: Type: text/plain, Size: 321 bytes --]

Hi all,

I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
Basically increase queue size can get better RX rate for my case.

Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?

Thanks,
Yajun



[-- Attachment #2: Type: text/html, Size: 2336 bytes --]

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
> I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
> Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
> Basically increase queue size can get better RX rate for my case.
> 
> Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?

Hi,
The VIRTIO 1.1 specification says the maximum number of descriptors is
32768 for both split and packed virtqueues.

The vhost kernel code seems to support 32768.

The 1024 limit is an implementation limit in QEMU. Increasing it would
require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
the size of arrays.

I can't think of a fundamental reason why QEMU needs to limit itself to
1024 descriptors. Raising the limit would require fixing up the code and
ensuring that live migration remains compatible with older versions of
QEMU.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Michael S. Tsirkin]

On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
> On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
> > I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
> > Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
> > Basically increase queue size can get better RX rate for my case.
> > 
> > Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
> 
> Hi,
> The VIRTIO 1.1 specification says the maximum number of descriptors is
> 32768 for both split and packed virtqueues.
> 
> The vhost kernel code seems to support 32768.
> 
> The 1024 limit is an implementation limit in QEMU. Increasing it would
> require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
> the size of arrays.
> 
> I can't think of a fundamental reason why QEMU needs to limit itself to
> 1024 descriptors. Raising the limit would require fixing up the code and
> ensuring that live migration remains compatible with older versions of
> QEMU.
> 
> Stefan

There's actually a reason for a limit: in theory the vq size
also sets a limit on the number of scatter/gather entries.
both QEMU and vhost can't handle a packet split over > 1k chunks.

We could add an extra limit for s/g size like block and scsi do,
this will need spec, guest and host side work.

-- 
MST

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

On Wed, Aug 05, 2020 at 08:13:29AM -0400, Michael S. Tsirkin wrote:
> On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
> > On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
> > > I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
> > > Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
> > > Basically increase queue size can get better RX rate for my case.
> > > 
> > > Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
> > 
> > Hi,
> > The VIRTIO 1.1 specification says the maximum number of descriptors is
> > 32768 for both split and packed virtqueues.
> > 
> > The vhost kernel code seems to support 32768.
> > 
> > The 1024 limit is an implementation limit in QEMU. Increasing it would
> > require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
> > the size of arrays.
> > 
> > I can't think of a fundamental reason why QEMU needs to limit itself to
> > 1024 descriptors. Raising the limit would require fixing up the code and
> > ensuring that live migration remains compatible with older versions of
> > QEMU.
> > 
> > Stefan
> 
> There's actually a reason for a limit: in theory the vq size
> also sets a limit on the number of scatter/gather entries.
> both QEMU and vhost can't handle a packet split over > 1k chunks.
> 
> We could add an extra limit for s/g size like block and scsi do,
> this will need spec, guest and host side work.

Interesting, thanks for explaining! This could be made explicit by
changing the QEMU code to:

include/hw/virtio/virtio.h:#define VIRTQUEUE_MAX_SIZE IOV_MAX

Looking more closely at the vhost kernel code I see that UIO_MAXIOV is
used in some places but not in vhost_vring_set_num() (ioctl
VHOST_SET_VRING_NUM). Is there a reason why UIO_MAXIOV isn't enforced
when the application sets the queue size?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Jason Wang]

On 2020/8/6 下午8:37, Stefan Hajnoczi wrote:
> On Wed, Aug 05, 2020 at 08:13:29AM -0400, Michael S. Tsirkin wrote:
>> On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
>>> On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
>>>> I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
>>>> Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
>>>> Basically increase queue size can get better RX rate for my case.
>>>>
>>>> Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
>>> Hi,
>>> The VIRTIO 1.1 specification says the maximum number of descriptors is
>>> 32768 for both split and packed virtqueues.
>>>
>>> The vhost kernel code seems to support 32768.
>>>
>>> The 1024 limit is an implementation limit in QEMU. Increasing it would
>>> require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
>>> the size of arrays.
>>>
>>> I can't think of a fundamental reason why QEMU needs to limit itself to
>>> 1024 descriptors. Raising the limit would require fixing up the code and
>>> ensuring that live migration remains compatible with older versions of
>>> QEMU.
>>>
>>> Stefan
>> There's actually a reason for a limit: in theory the vq size
>> also sets a limit on the number of scatter/gather entries.
>> both QEMU and vhost can't handle a packet split over > 1k chunks.
>>
>> We could add an extra limit for s/g size like block and scsi do,
>> this will need spec, guest and host side work.
> Interesting, thanks for explaining! This could be made explicit by
> changing the QEMU code to:
>
> include/hw/virtio/virtio.h:#define VIRTQUEUE_MAX_SIZE IOV_MAX
>
> Looking more closely at the vhost kernel code I see that UIO_MAXIOV is
> used in some places but not in vhost_vring_set_num() (ioctl
> VHOST_SET_VRING_NUM). Is there a reason why UIO_MAXIOV isn't enforced
> when the application sets the queue size?


Actually three things:

1) queue size
2) #descriptors in a list
3) IOV size

Spec limit the 2) to 1) but 2) may not equal to 3).

So enforcing UIO_MAXIOV can not solve the problem completely.

For vhost-net, it depends on socket to build skb which requires an iov 
array to work. We need remove this limitation by:

- build skb by vhost-net itself
- do piecewise copying

Then we're not limited with #iov and more and support up to what spec 
supports.

Thanks


>
> Stefan

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2845 bytes --]

On Fri, Aug 07, 2020 at 11:35:08AM +0800, Jason Wang wrote:
> 
> On 2020/8/6 下午8:37, Stefan Hajnoczi wrote:
> > On Wed, Aug 05, 2020 at 08:13:29AM -0400, Michael S. Tsirkin wrote:
> > > On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
> > > > On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
> > > > > I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
> > > > > Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
> > > > > Basically increase queue size can get better RX rate for my case.
> > > > > 
> > > > > Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
> > > > Hi,
> > > > The VIRTIO 1.1 specification says the maximum number of descriptors is
> > > > 32768 for both split and packed virtqueues.
> > > > 
> > > > The vhost kernel code seems to support 32768.
> > > > 
> > > > The 1024 limit is an implementation limit in QEMU. Increasing it would
> > > > require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
> > > > the size of arrays.
> > > > 
> > > > I can't think of a fundamental reason why QEMU needs to limit itself to
> > > > 1024 descriptors. Raising the limit would require fixing up the code and
> > > > ensuring that live migration remains compatible with older versions of
> > > > QEMU.
> > > > 
> > > > Stefan
> > > There's actually a reason for a limit: in theory the vq size
> > > also sets a limit on the number of scatter/gather entries.
> > > both QEMU and vhost can't handle a packet split over > 1k chunks.
> > > 
> > > We could add an extra limit for s/g size like block and scsi do,
> > > this will need spec, guest and host side work.
> > Interesting, thanks for explaining! This could be made explicit by
> > changing the QEMU code to:
> > 
> > include/hw/virtio/virtio.h:#define VIRTQUEUE_MAX_SIZE IOV_MAX
> > 
> > Looking more closely at the vhost kernel code I see that UIO_MAXIOV is
> > used in some places but not in vhost_vring_set_num() (ioctl
> > VHOST_SET_VRING_NUM). Is there a reason why UIO_MAXIOV isn't enforced
> > when the application sets the queue size?
> 
> 
> Actually three things:
> 
> 1) queue size
> 2) #descriptors in a list
> 3) IOV size
> 
> Spec limit the 2) to 1) but 2) may not equal to 3).
> 
> So enforcing UIO_MAXIOV can not solve the problem completely.
> 
> For vhost-net, it depends on socket to build skb which requires an iov array
> to work. We need remove this limitation by:
> 
> - build skb by vhost-net itself
> - do piecewise copying
> 
> Then we're not limited with #iov and more and support up to what spec
> supports.

If I understand correctly, you are saying vhost_net.ko does not support
more than UIO_MAXIOV descriptors today but it could be fixed as you
described.

Thanks!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Jason Wang]

On 2020/8/7 下午5:59, Stefan Hajnoczi wrote:
> On Fri, Aug 07, 2020 at 11:35:08AM +0800, Jason Wang wrote:
>> On 2020/8/6 下午8:37, Stefan Hajnoczi wrote:
>>> On Wed, Aug 05, 2020 at 08:13:29AM -0400, Michael S. Tsirkin wrote:
>>>> On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
>>>>> On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
>>>>>> I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
>>>>>> Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
>>>>>> Basically increase queue size can get better RX rate for my case.
>>>>>>
>>>>>> Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
>>>>> Hi,
>>>>> The VIRTIO 1.1 specification says the maximum number of descriptors is
>>>>> 32768 for both split and packed virtqueues.
>>>>>
>>>>> The vhost kernel code seems to support 32768.
>>>>>
>>>>> The 1024 limit is an implementation limit in QEMU. Increasing it would
>>>>> require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
>>>>> the size of arrays.
>>>>>
>>>>> I can't think of a fundamental reason why QEMU needs to limit itself to
>>>>> 1024 descriptors. Raising the limit would require fixing up the code and
>>>>> ensuring that live migration remains compatible with older versions of
>>>>> QEMU.
>>>>>
>>>>> Stefan
>>>> There's actually a reason for a limit: in theory the vq size
>>>> also sets a limit on the number of scatter/gather entries.
>>>> both QEMU and vhost can't handle a packet split over > 1k chunks.
>>>>
>>>> We could add an extra limit for s/g size like block and scsi do,
>>>> this will need spec, guest and host side work.
>>> Interesting, thanks for explaining! This could be made explicit by
>>> changing the QEMU code to:
>>>
>>> include/hw/virtio/virtio.h:#define VIRTQUEUE_MAX_SIZE IOV_MAX
>>>
>>> Looking more closely at the vhost kernel code I see that UIO_MAXIOV is
>>> used in some places but not in vhost_vring_set_num() (ioctl
>>> VHOST_SET_VRING_NUM). Is there a reason why UIO_MAXIOV isn't enforced
>>> when the application sets the queue size?
>>
>> Actually three things:
>>
>> 1) queue size
>> 2) #descriptors in a list
>> 3) IOV size
>>
>> Spec limit the 2) to 1) but 2) may not equal to 3).
>>
>> So enforcing UIO_MAXIOV can not solve the problem completely.
>>
>> For vhost-net, it depends on socket to build skb which requires an iov array
>> to work. We need remove this limitation by:
>>
>> - build skb by vhost-net itself
>> - do piecewise copying
>>
>> Then we're not limited with #iov and more and support up to what spec
>> supports.
> If I understand correctly, you are saying vhost_net.ko does not support
> more than UIO_MAXIOV descriptors today but it could be fixed as you
> described.


Yes, but it needs major refactoring on vhost_net.

thanks


>
> Thanks!
>
> Stefan

Re: Any reason VIRTQUEUE_MAX_SIZE is 1024? Can we increase this limit?

[Michael S. Tsirkin]

On Thu, Aug 06, 2020 at 01:37:08PM +0100, Stefan Hajnoczi wrote:
> On Wed, Aug 05, 2020 at 08:13:29AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Aug 05, 2020 at 01:11:07PM +0100, Stefan Hajnoczi wrote:
> > > On Thu, Jul 30, 2020 at 07:46:09AM +0000, Yajun Wu wrote:
> > > > I'm doing iperf test on VIRTIO net through vhost-user(HW VDPA).
> > > > Find maximal acceptable tx_queue_size/rx_queue_size is 1024.
> > > > Basically increase queue size can get better RX rate for my case.
> > > > 
> > > > Can we increase the limit(VIRTQUEUE_MAX_SIZE) to 8192 to possibly gain better performance?
> > > 
> > > Hi,
> > > The VIRTIO 1.1 specification says the maximum number of descriptors is
> > > 32768 for both split and packed virtqueues.
> > > 
> > > The vhost kernel code seems to support 32768.
> > > 
> > > The 1024 limit is an implementation limit in QEMU. Increasing it would
> > > require QEMU code changes. For example, VIRTQUEUE_MAX_SIZE is used as
> > > the size of arrays.
> > > 
> > > I can't think of a fundamental reason why QEMU needs to limit itself to
> > > 1024 descriptors. Raising the limit would require fixing up the code and
> > > ensuring that live migration remains compatible with older versions of
> > > QEMU.
> > > 
> > > Stefan
> > 
> > There's actually a reason for a limit: in theory the vq size
> > also sets a limit on the number of scatter/gather entries.
> > both QEMU and vhost can't handle a packet split over > 1k chunks.
> > 
> > We could add an extra limit for s/g size like block and scsi do,
> > this will need spec, guest and host side work.
> 
> Interesting, thanks for explaining! This could be made explicit by
> changing the QEMU code to:
> 
> include/hw/virtio/virtio.h:#define VIRTQUEUE_MAX_SIZE IOV_MAX
> 
> Looking more closely at the vhost kernel code I see that UIO_MAXIOV is
> used in some places but not in vhost_vring_set_num() (ioctl
> VHOST_SET_VRING_NUM). Is there a reason why UIO_MAXIOV isn't enforced
> when the application sets the queue size?
> 
> Stefan


Backends such as vhost-user can handle > iov max. Devices such
as scsi and block have a limit for s/g separate from vq size.

-- 
MST