Can we increase vring size over 1024?

Can we increase vring size over 1024?

[Gonglei (Arei)]

Hi Michael & all,

Michael, you made a presentation about the virto 1.1's new features in KVM Forum last week.
That's wonderful!

And I'd like to know can we increase vring size over 1024, such as 4096 or 8192?

My colleage had asked the same question in 2014, but she didn't get a definite answare,
So, I want to rewake up the dissusstion about this. Becase for the virtio-crypto device,
I also need to increase the vring size to get better performance and thoughput, but the Qemu
side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.

 [QA-virtio]:Why vring size is limited to 1024?
http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-1024-td292450.html

Avi Kivity said that google cloud exposed the vring size to 16k.

Regards,
-Gonglei

Re: [Qemu-devel] Can we increase vring size over 1024?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1797 bytes --]

On Fri, Sep 02, 2016 at 06:55:35AM +0000, Gonglei (Arei) wrote:
> Michael, you made a presentation about the virto 1.1's new features in KVM Forum last week.
> That's wonderful!
> 
> And I'd like to know can we increase vring size over 1024, such as 4096 or 8192?
> 
> My colleage had asked the same question in 2014, but she didn't get a definite answare,
> So, I want to rewake up the dissusstion about this. Becase for the virtio-crypto device,
> I also need to increase the vring size to get better performance and thoughput, but the Qemu
> side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.
> 
>  [QA-virtio]:Why vring size is limited to 1024?
> http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-1024-td292450.html
> 
> Avi Kivity said that google cloud exposed the vring size to 16k.

VIRTIO 1.0 allows the driver to choose the ring size (the device
provides the maximum value), so in theory QEMU could offer large maximum
values to VIRTIO 1.0 guests.  But existing drivers may use the device's
maximum value (~16k) instead of choosing a reasonable smaller value
(~1k), leading to much larger virtqueues in cases that don't need them.

I'd like to understand the root cause for the performance issue you see
with virtio-net 1024 ring size.  Which queue is too small: rx or tx?

Are rx packets dropped because 1) the guest is not servicing the
virtqueue quickly enough or 2) is the physical NIC delivering more
packets than the rx virtqueue size?

In case #1 the guest needs to move packets from the rx virtqueue to a
software queue as quickly as possible.  The virtqueue is not designed to
be the incoming packet queue for the application.

In case #2 it may be possible to adjust code in the host to handle this
gracefully.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

答复: Can we increase vring size over 1024?

[Lilijun (Jerry)]

Hi Stefan,

In our test case, the virtio TX queue is not big enough. The packets throughput sent by application in VM is not stable, Sometimes it become very large and exceed the service capability of the backend virtual switch, then they were dropped unfortunately.

To enlarge application and virtual switch's buffer ring may result other difficult problems, So we'd like to increase the virtio TX queue size to 4K or 8K.

Thanks.

-----邮件原件-----
发件人: Stefan Hajnoczi [mailto:stefanha@gmail.com] 
发送时间: 2016年9月2日 21:40
收件人: Gonglei (Arei)
抄送: mst@redhat.com; stefanha@redhat.com; jasowang@redhat.com; virtio-dev@lists.oasis-open.org; Huangpeng (Peter); qemu-devel@nongnu.org; kvm@vger.kernel.org; Lilijun (Jerry)
主题: Re: [Qemu-devel] Can we increase vring size over 1024?

On Fri, Sep 02, 2016 at 06:55:35AM +0000, Gonglei (Arei) wrote:
> Michael, you made a presentation about the virto 1.1's new features in KVM Forum last week.
> That's wonderful!
> 
> And I'd like to know can we increase vring size over 1024, such as 4096 or 8192?
> 
> My colleage had asked the same question in 2014, but she didn't get a 
> definite answare, So, I want to rewake up the dissusstion about this. 
> Becase for the virtio-crypto device, I also need to increase the vring 
> size to get better performance and thoughput, but the Qemu side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.
> 
>  [QA-virtio]:Why vring size is limited to 1024?
> http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-10
> 24-td292450.html
> 
> Avi Kivity said that google cloud exposed the vring size to 16k.

VIRTIO 1.0 allows the driver to choose the ring size (the device provides the maximum value), so in theory QEMU could offer large maximum values to VIRTIO 1.0 guests.  But existing drivers may use the device's maximum value (~16k) instead of choosing a reasonable smaller value (~1k), leading to much larger virtqueues in cases that don't need them.

I'd like to understand the root cause for the performance issue you see with virtio-net 1024 ring size.  Which queue is too small: rx or tx?

Are rx packets dropped because 1) the guest is not servicing the virtqueue quickly enough or 2) is the physical NIC delivering more packets than the rx virtqueue size?

In case #1 the guest needs to move packets from the rx virtqueue to a software queue as quickly as possible.  The virtqueue is not designed to be the incoming packet queue for the application.

In case #2 it may be possible to adjust code in the host to handle this gracefully.

Stefan

Re: 答复: [Qemu-devel] Can we increase vring size over 1024?

[Michael S. Tsirkin]

On Mon, Sep 05, 2016 at 03:21:44AM +0000, Lilijun (Jerry) wrote:
> Hi Stefan,
> 
> In our test case, the virtio TX queue is not big enough. The packets throughput sent by application in VM is not stable, Sometimes it become very large and exceed the service capability of the backend virtual switch, then they were dropped unfortunately.
> 
> To enlarge application and virtual switch's buffer ring may result other difficult problems, So we'd like to increase the virtio TX queue size to 4K or 8K.
> 
> Thanks.


Imagine a packet at the tail of the 8K queue.  How long will it take for
it to be transmitted?  If we do transmit this huge batch of 8K packets
in one go, how likely are we to cause jitter and latency for other
network users?

Looks like classical bufferbloat to me.

So again, it's a policy question and it does make sense to allow this in
future spec versions, but I can't say this is my top priority. You will
have to go ahead and design an interface to split s/g size and ring size
yourself if you are so inclined.

> -----邮件原件-----
> 发件人: Stefan Hajnoczi [mailto:stefanha@gmail.com] 
> 发送时间: 2016年9月2日 21:40
> 收件人: Gonglei (Arei)
> 抄送: mst@redhat.com; stefanha@redhat.com; jasowang@redhat.com; virtio-dev@lists.oasis-open.org; Huangpeng (Peter); qemu-devel@nongnu.org; kvm@vger.kernel.org; Lilijun (Jerry)
> 主题: Re: [Qemu-devel] Can we increase vring size over 1024?
> 
> On Fri, Sep 02, 2016 at 06:55:35AM +0000, Gonglei (Arei) wrote:
> > Michael, you made a presentation about the virto 1.1's new features in KVM Forum last week.
> > That's wonderful!
> > 
> > And I'd like to know can we increase vring size over 1024, such as 4096 or 8192?
> > 
> > My colleage had asked the same question in 2014, but she didn't get a 
> > definite answare, So, I want to rewake up the dissusstion about this. 
> > Becase for the virtio-crypto device, I also need to increase the vring 
> > size to get better performance and thoughput, but the Qemu side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.
> > 
> >  [QA-virtio]:Why vring size is limited to 1024?
> > http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-10
> > 24-td292450.html
> > 
> > Avi Kivity said that google cloud exposed the vring size to 16k.
> 
> VIRTIO 1.0 allows the driver to choose the ring size (the device provides the maximum value), so in theory QEMU could offer large maximum values to VIRTIO 1.0 guests.  But existing drivers may use the device's maximum value (~16k) instead of choosing a reasonable smaller value (~1k), leading to much larger virtqueues in cases that don't need them.
> 
> I'd like to understand the root cause for the performance issue you see with virtio-net 1024 ring size.  Which queue is too small: rx or tx?
> 
> Are rx packets dropped because 1) the guest is not servicing the virtqueue quickly enough or 2) is the physical NIC delivering more packets than the rx virtqueue size?
> 
> In case #1 the guest needs to move packets from the rx virtqueue to a software queue as quickly as possible.  The virtqueue is not designed to be the incoming packet queue for the application.
> 
> In case #2 it may be possible to adjust code in the host to handle this gracefully.
> 
> Stefan

Re: 答复: [Qemu-devel] Can we increase vring size over 1024?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

On Mon, Sep 05, 2016 at 03:21:44AM +0000, Lilijun (Jerry) wrote:
> In our test case, the virtio TX queue is not big enough. The packets throughput sent by application in VM is not stable, Sometimes it become very large and exceed the service capability of the backend virtual switch, then they were dropped unfortunately.
> 
> To enlarge application and virtual switch's buffer ring may result other difficult problems, So we'd like to increase the virtio TX queue size to 4K or 8K.

Why is it okay to enlarge the virtio-net tx queue but not the
application or switch queue?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: Can we increase vring size over 1024?

[Michael S. Tsirkin]

On Fri, Sep 02, 2016 at 06:55:35AM +0000, Gonglei (Arei) wrote:
> Hi Michael & all,
> 
> Michael, you made a presentation about the virto 1.1's new features in KVM Forum last week.
> That's wonderful!
> 
> And I'd like to know can we increase vring size over 1024, such as 4096 or 8192?
> 
> My colleage had asked the same question in 2014, but she didn't get a definite answare,
> So, I want to rewake up the dissusstion about this. Becase for the virtio-crypto device,
> I also need to increase the vring size to get better performance and thoughput, but the Qemu
> side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.
> 
>  [QA-virtio]:Why vring size is limited to 1024?
> http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-1024-td292450.html
> 
> Avi Kivity said that google cloud exposed the vring size to 16k.
> 
> Regards,
> -Gonglei

Fundamentally, the reason is that the ring size
currently also sets the max s/g list length, and linux
hosts can't support bigger lists.

We should fix that in 1.1.

-- 
MST

RE: Can we increase vring size over 1024?

[Gonglei (Arei)]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Saturday, September 03, 2016 1:33 AM
> To: Gonglei (Arei)
> Cc: stefanha@redhat.com; jasowang@redhat.com; kvm@vger.kernel.org;
> virtio-dev@lists.oasis-open.org; qemu-devel@nongnu.org; Lilijun (Jerry);
> Huangpeng (Peter)
> Subject: Re: Can we increase vring size over 1024?
> 
> On Fri, Sep 02, 2016 at 06:55:35AM +0000, Gonglei (Arei) wrote:
> > Hi Michael & all,
> >
> > Michael, you made a presentation about the virto 1.1's new features in KVM
> Forum last week.
> > That's wonderful!
> >
> > And I'd like to know can we increase vring size over 1024, such as 4096 or
> 8192?
> >
> > My colleage had asked the same question in 2014, but she didn't get a
> definite answare,
> > So, I want to rewake up the dissusstion about this. Becase for the
> virtio-crypto device,
> > I also need to increase the vring size to get better performance and
> thoughput, but the Qemu
> > side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.
> >
> >  [QA-virtio]:Why vring size is limited to 1024?
> >
> http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-1024-td2
> 92450.html
> >
> > Avi Kivity said that google cloud exposed the vring size to 16k.
> >
> > Regards,
> > -Gonglei
> 
> Fundamentally, the reason is that the ring size
> currently also sets the max s/g list length, and linux
> hosts can't support bigger lists.
> 
But I don't think this is a problem.
Vring is just a container, we can say the max request's length is 1024, but the capacity of container
shouldn't be the length of max request. For example, we can put 4K requests with one s/g list
into vring at one time if the vring size is 4096, and 4 requests with 1024 s/g list into vring at one time.
Ignoring the indirect table support. Am I right?

> We should fix that in 1.1.
> 
Sounds good!

> --
> MST

Regards,
-Gonglei

Re: Can we increase vring size over 1024?

[Michael S. Tsirkin]

On Sat, Sep 03, 2016 at 04:20:47AM +0000, Gonglei (Arei) wrote:
> 
> 
> > -----Original Message-----
> > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > Sent: Saturday, September 03, 2016 1:33 AM
> > To: Gonglei (Arei)
> > Cc: stefanha@redhat.com; jasowang@redhat.com; kvm@vger.kernel.org;
> > virtio-dev@lists.oasis-open.org; qemu-devel@nongnu.org; Lilijun (Jerry);
> > Huangpeng (Peter)
> > Subject: Re: Can we increase vring size over 1024?
> > 
> > On Fri, Sep 02, 2016 at 06:55:35AM +0000, Gonglei (Arei) wrote:
> > > Hi Michael & all,
> > >
> > > Michael, you made a presentation about the virto 1.1's new features in KVM
> > Forum last week.
> > > That's wonderful!
> > >
> > > And I'd like to know can we increase vring size over 1024, such as 4096 or
> > 8192?
> > >
> > > My colleage had asked the same question in 2014, but she didn't get a
> > definite answare,
> > > So, I want to rewake up the dissusstion about this. Becase for the
> > virtio-crypto device,
> > > I also need to increase the vring size to get better performance and
> > thoughput, but the Qemu
> > > side limit the thought as VIRTQUEUE_MAX_SIZE is 1024.
> > >
> > >  [QA-virtio]:Why vring size is limited to 1024?
> > >
> > http://qemu.11.n7.nabble.com/QA-virtio-Why-vring-size-is-limited-to-1024-td2
> > 92450.html
> > >
> > > Avi Kivity said that google cloud exposed the vring size to 16k.
> > >
> > > Regards,
> > > -Gonglei
> > 
> > Fundamentally, the reason is that the ring size
> > currently also sets the max s/g list length, and linux
> > hosts can't support bigger lists.
> > 
> But I don't think this is a problem.
> Vring is just a container, we can say the max request's length is 1024, but the capacity of container
> shouldn't be the length of max request. For example, we can put 4K requests with one s/g list
> into vring at one time if the vring size is 4096, and 4 requests with 1024 s/g list into vring at one time.
> Ignoring the indirect table support. Am I right?

We'd have to change the spec for that. At the moment spec says
they are the same. Features that link descriptors when they
are used, like mergeable buffers, would also need to obey the
new limit. When vring size is used, both fall out naturally
(except with indirect descriptors).

> > We should fix that in 1.1.
> > 
> Sounds good!
> 
> > --
> > MST
> 
> Regards,
> -Gonglei