Re: [Qemu-devel] Block I/O optimizations

Re: [Qemu-devel] Block I/O optimizations

[Stefan Hajnoczi]

On Thu, Feb 28, 2013 at 08:20:08PM +0200, Abel Gordon wrote:
> Stefan Hajnoczi <stefanha@gmail.com> wrote on 28/02/2013 04:43:04 PM:
> > I think extending and tuning the existing mechanisms is the way to go.
> > I don't see obvious advantages other than reducing context switches.
> 
> Maybe it is worth checking...
> We did experiments using vhost-net and vhost-blk. We measured and compared
> the traditional model (kernel thread per VM/virtual device) to the
> shared-thread model with fine-grained I/O scheduling (single kernel thread
> used to serve multiple VMs). We noticed improvements up-to 2.5x
> in throughput and almost half the latency when running up-to 14 VMs.

Can you post patches?

Also, I wonder if you have time to do a presentation/discussion session
so we can get the ball rolling and more people exposed to your approach.
There is a weekly QEMU Community Call which we can use as the forum.

The reason I have been skeptical is that prototyping radical changes
often involves rewriting or bypassing code.  These "accidental" changes
can impact performance too.  We need to understand where to attribute
the performance improvements.

Stefan

Re: [Qemu-devel] Block I/O optimizations

[Abel Gordon]

Stefan Hajnoczi <stefanha@gmail.com>  wrote on 01/03/2013 12:54:54 PM:

> On Thu, Feb 28, 2013 at 08:20:08PM +0200, Abel Gordon wrote:
> > Stefan Hajnoczi <stefanha@gmail.com> wrote on 28/02/2013 04:43:04 PM:
> > > I think extending and tuning the existing mechanisms is the way to
go.
> > > I don't see obvious advantages other than reducing context switches.
> >
> > Maybe it is worth checking...
> > We did experiments using vhost-net and vhost-blk. We measured and
compared
> > the traditional model (kernel thread per VM/virtual device) to the
> > shared-thread model with fine-grained I/O scheduling (single kernel
thread
> > used to serve multiple VMs). We noticed improvements up-to 2.5x
> > in throughput and almost half the latency when running up-to 14 VMs.
>
> Can you post patches?

We will publish the code soon but note the patches are for vhost
kernel back-end and not for the qemu user-space back-end.

> Also, I wonder if you have time to do a presentation/discussion session
> so we can get the ball rolling and more people exposed to your approach.
> There is a weekly QEMU Community Call which we can use as the forum.

Sure. I'll send you a separate email to schedule the
presentation/discussion.

Re: [Qemu-devel] Block I/O optimizations

[Stefan Hajnoczi]

On Sun, Mar 3, 2013 at 10:35 AM, Abel Gordon <ABELG@il.ibm.com> wrote:
>
>
> Stefan Hajnoczi <stefanha@gmail.com>  wrote on 01/03/2013 12:54:54 PM:
>
>> On Thu, Feb 28, 2013 at 08:20:08PM +0200, Abel Gordon wrote:
>> > Stefan Hajnoczi <stefanha@gmail.com> wrote on 28/02/2013 04:43:04 PM:
>> > > I think extending and tuning the existing mechanisms is the way to
> go.
>> > > I don't see obvious advantages other than reducing context switches.
>> >
>> > Maybe it is worth checking...
>> > We did experiments using vhost-net and vhost-blk. We measured and
> compared
>> > the traditional model (kernel thread per VM/virtual device) to the
>> > shared-thread model with fine-grained I/O scheduling (single kernel
> thread
>> > used to serve multiple VMs). We noticed improvements up-to 2.5x
>> > in throughput and almost half the latency when running up-to 14 VMs.
>>
>> Can you post patches?
>
> We will publish the code soon but note the patches are for vhost
> kernel back-end and not for the qemu user-space back-end.

That's fine.  The only difference the codebase makes is which mailing list:

 * qemu-devel@nongnu.org - QEMU userspace
 * kvm@vger.kernel.org - kvm kernel module
 * virtualization@lists.linuxfoundation.org - broader scope Linux
kernel virtualization (vhost, virtio, hyperv drivers, etc)

Stefan

Re: [Qemu-devel] Block I/O optimizations

[Michael S. Tsirkin]

On Sun, Mar 03, 2013 at 11:35:27AM +0200, Abel Gordon wrote:
> 
> 
> Stefan Hajnoczi <stefanha@gmail.com>  wrote on 01/03/2013 12:54:54 PM:
> 
> > On Thu, Feb 28, 2013 at 08:20:08PM +0200, Abel Gordon wrote:
> > > Stefan Hajnoczi <stefanha@gmail.com> wrote on 28/02/2013 04:43:04 PM:
> > > > I think extending and tuning the existing mechanisms is the way to
> go.
> > > > I don't see obvious advantages other than reducing context switches.
> > >
> > > Maybe it is worth checking...
> > > We did experiments using vhost-net and vhost-blk. We measured and
> compared
> > > the traditional model (kernel thread per VM/virtual device) to the
> > > shared-thread model with fine-grained I/O scheduling (single kernel
> thread
> > > used to serve multiple VMs). We noticed improvements up-to 2.5x
> > > in throughput and almost half the latency when running up-to 14 VMs.
> >
> > Can you post patches?
> 
> We will publish the code soon but note the patches are for vhost
> kernel back-end and not for the qemu user-space back-end.

Yes, this is one of the fields where the asynchronous interface
of vhost could be helpful, abstracting the threading model
away from the application.

The main challenge with sharing threads is handling things like cpu
limits and swap access. They normally send the current thread to sleep
and let another thread run, but if VMs share a thread, this becomes a
problem.  Could be solvable by detecting such conditions and moving
other VMs to per-VM threads, but I haven't seen such patches yet.


> > Also, I wonder if you have time to do a presentation/discussion session
> > so we can get the ball rolling and more people exposed to your approach.
> > There is a weekly QEMU Community Call which we can use as the forum.
> 
> Sure. I'll send you a separate email to schedule the
> presentation/discussion.
> 

Re: [Qemu-devel] Block I/O optimizations

[Stefan Hajnoczi]

On Thu, Feb 21, 2013 at 9:11 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Mon, Feb 18, 2013 at 7:19 PM, Loic Dachary <loic@dachary.org> wrote:
>> I recently tried to figure out the best and easiest ways to increase block I/O performances with qemu. Not being a qemu expert, I expected to find a few optimization tricks. Much to my surprise, it appears that there are many significant improvements being worked on. This is excellent news :-)
>>
>> However, I'm not sure I understand how they all fit together. It's probably quite obvious from the developer point of view but I would very much appreciate an overview of how dataplane, vhost-blk, ELVIS etc. should be used or developed to maximize I/O performances. Are there documents I should read ? If not, would someone be willing to share bits of wisdom ?
>
> Hi Loic,
> There will be more information on dataplane shortly.  I'll write up a
> blog post and share the link with you.

Hi Loic,
Here is an overview of virtio-blk data plane and how to use it:
http://blog.vmsplice.net/2013/03/new-in-qemu-14-high-performance-virtio.html

Stefan

Re: [Qemu-devel] Block I/O optimizations

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

Hi Stefan,

This is excellent ! Thank you very much :-)

Cheers

On 03/13/2013 04:21 PM, Stefan Hajnoczi wrote:
> On Thu, Feb 21, 2013 at 9:11 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Mon, Feb 18, 2013 at 7:19 PM, Loic Dachary <loic@dachary.org> wrote:
>>> I recently tried to figure out the best and easiest ways to increase block I/O performances with qemu. Not being a qemu expert, I expected to find a few optimization tricks. Much to my surprise, it appears that there are many significant improvements being worked on. This is excellent news :-)
>>>
>>> However, I'm not sure I understand how they all fit together. It's probably quite obvious from the developer point of view but I would very much appreciate an overview of how dataplane, vhost-blk, ELVIS etc. should be used or developed to maximize I/O performances. Are there documents I should read ? If not, would someone be willing to share bits of wisdom ?
>>
>> Hi Loic,
>> There will be more information on dataplane shortly.  I'll write up a
>> blog post and share the link with you.
> 
> Hi Loic,
> Here is an overview of virtio-blk data plane and how to use it:
> http://blog.vmsplice.net/2013/03/new-in-qemu-14-high-performance-virtio.html
> 
> Stefan
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Block I/O optimizations

[Abel Gordon]

qemu-devel-bounces+abelg=il.ibm.com@nongnu.org wrote on 03/03/2013 11:35:27
AM:

> > Also, I wonder if you have time to do a presentation/discussion session
> > so we can get the ball rolling and more people exposed to your
approach.
> > There is a weekly QEMU Community Call which we can use as the forum.
>
> Sure. I'll send you a separate email to schedule the
> presentation/discussion.

We just posted a technical report that describes the design and evaluation
of the work we did to improve virtual net/block I/O scalability and
performance based on vhost and hosting multiple KVM guests.
You can find the report here: http://goo.gl/RKppy

(full link:
http://domino.research.ibm.com/library/cyberdig.nsf/1e4115aea78b6e7c85256b360066f0d4/479e3578ed05bfac85257b4200427735!OpenDocument&Highlight=0,h-0319
)

I can present this work in the next QEMU community call.

Regards,
Abel.

Re: [Qemu-devel] Block I/O optimizations

[Anthony Liguori]

Abel Gordon <ABELG@il.ibm.com> writes:

> qemu-devel-bounces+abelg=il.ibm.com@nongnu.org wrote on 03/03/2013 11:35:27
> AM:
>
> We just posted a technical report that describes the design and evaluation
> of the work we did to improve virtual net/block I/O scalability and
> performance based on vhost and hosting multiple KVM guests.
> You can find the report here: http://goo.gl/RKppy
>
> (full link:
> http://domino.research.ibm.com/library/cyberdig.nsf/1e4115aea78b6e7c85256b360066f0d4/479e3578ed05bfac85257b4200427735!OpenDocument&Highlight=0,h-0319
> )
>
> I can present this work in the next QEMU community call.

That would be great!  Thanks Abel.

Regards,

Anthony Liguori

>
> Regards,
> Abel.

Re: [Qemu-devel] Block I/O optimizations

[Liu Yuan]

On 04/04/2013 04:36 PM, Abel Gordon wrote:
> We just posted a technical report that describes the design and evaluation
> of the work we did to improve virtual net/block I/O scalability and
> performance based on vhost and hosting multiple KVM guests.
> You can find the report here: http://goo.gl/RKppy
> 
> (full link:
> http://domino.research.ibm.com/library/cyberdig.nsf/1e4115aea78b6e7c85256b360066f0d4/479e3578ed05bfac85257b4200427735!OpenDocument&Highlight=0,h-0319
> )
> 
> I can present this work in the next QEMU community call.

Hi Abel,

   Could you share the presentation to me? I am very interested in your
works and I once have had done some works on vhost_blk. (I can't attend
community call)

Thanks,
Yuan