Virtio Block Device Queue Depth

Virtio Block Device Queue Depth

[George Bottas]

Hi list,

I have a question regarding changing the queue size that is set in virtio_blk_init(). The current value is 128, which results in setting the queue depth in the Windows guest device to 8. Does anyone know if changing this value to defined maximum (1024) going to result in any issues? NB, we have also increased the thread pool accordingly. Our testing so far has produced a situation where initially we are seeing very high read request rates (1500+), and then at some point the reads from the Windows guest become serialized, i.e. the Windows guest synchronously sends one READ to the host, waits for it to complete, sends the next one, etc.  Has this ever been seen anywhere else? 

Any help on this would be appreciated.

Thanks,
George

Re: Virtio Block Device Queue Depth

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 5:48 PM, George Bottas <gbottas@juniper.net> wrote:
> I have a question regarding changing the queue size that is set in virtio_blk_init(). The current value is 128, which results in setting the queue depth in the Windows guest device to 8. Does anyone know if changing this value to defined maximum (1024) going to result in any issues? NB, we have also increased the thread pool accordingly. Our testing so far has produced a situation where initially we are seeing very high read request rates (1500+), and then at some point the reads from the Windows guest become serialized, i.e. the Windows guest synchronously sends one READ to the host, waits for it to complete, sends the next one, etc.  Has this ever been seen anywhere else?
>
> Any help on this would be appreciated.

If you are really limited to 8 requests then the issue is not the
number of vring descriptors since recent virtio-blk has the indirect
vring feature enabled.  The indirect vring feature allows the guest to
send 128 parallel requests - much higher than the 8 you mentioned from
Windows.

CCing Vadim who maintains the Windows virtio-blk driver.

Stefan

Re: Virtio Block Device Queue Depth

[Vadim Rozenfeld]

On Friday, March 09, 2012 11:56:36 AM you wrote:
> On Thu, Mar 8, 2012 at 5:48 PM, George Bottas <gbottas@juniper.net> wrote:
> > I have a question regarding changing the queue size that is set in
> > virtio_blk_init(). The current value is 128, which results in setting
> > the queue depth in the Windows guest device to 8. Does anyone know if
Actually, it's 6 128/(16+1) - 1
> > changing this value to defined maximum (1024) going to result in any
In my experience, enalarging the queueu size doen't leas to any 
significant performance improvement 
> > issues? NB, we have also increased the thread pool accordingly. Our
> > testing so far has produced a situation where initially we are seeing
> > very high read request rates (1500+), and then at some point the reads
> > from the Windows guest become serialized, i.e. the Windows guest
> > synchronously sends one READ to the host, waits for it to complete,
> > sends the next one, etc.  Has this ever been seen anywhere else?
On Win2K3 and higher, viostor operats in full-duplex mode. Which means
that it must be able to process several requests simulteniousle. But
if you have 6 requests pending (processing), storport driver will throttle
incomming requestss. 
> > 
> > Any help on this would be appreciated.
> 
> If you are really limited to 8 requests then the issue is not the
> number of vring descriptors since recent virtio-blk has the indirect
> vring feature enabled.  The indirect vring feature allows the guest to
> send 128 parallel requests - much higher than the 8 you mentioned from
> Windows.
It's true. However, I cannot say that indirect buffers will automaticaly
give you a better performance. Inderect buffers should work faster for
peretty big data chanks (128K..256K), while for the normal load (4K..64K)
they give you almost the same numbers. 
Vadim.
> 
> CCing Vadim who maintains the Windows virtio-blk driver.
> 
> Stefan