[Qemu-devel] Linux multiqueue block layer thoughts

[Qemu-devel] Linux multiqueue block layer thoughts

[Stefan Hajnoczi]

I finally got around to reading the Linux multiqueue block layer paper
and wanted to share some thoughts about how it relates to QEMU and
dataplane/QContext:
http://kernel.dk/blk-mq.pdf

I think Jens has virtio-blk multiqueue patches.  So let's imagine that
the virtio-blk device has multiple virtqueues.  (virtio-scsi is
already multiqueue BTW.)

The paper focusses on two queue mappings: 1 queue per core and 1 queue
per node.  In both cases the idea is to keep the block I/O code path
localized.  This makes block I/O scale as the number of CPUs
increases.

In QEMU we'd want to set up a mapping for the virtio-blk mq device:
each guest vcpu or guest node has a virtio-blk virtqueue which is
serviced by a dataplane/QContext thread.

QEMU would then process requests across these queues in parallel,
although currently BlockDriverState is not thread-safe.  At least for
raw we should be able to submit requests in parallel from QEMU.

Unfortunately there are some complications in the QEMU block layer:
QEMU's own accounting, request tracking, and throttling features are
global.  We'd need to eventually do something similar to the
multiqueue block layer changes in the kernel to detangle this state.

Doing multiqueue for image formats is much more challenging - we'd
have to tackle thread-safety in qcow2 and friends.  For network block
drivers like Gluster or NBD it's also not 100% clear what the best
approach is.  But I think the target here is local SSDs that are
capable of high IOPs together with an SMP guest.

At the end of all this we'd arrive at the following architecture:
1. Guest virtio device has multiple queues (1 per node or vcpu).
2. QEMU has multiple dataplane/QContext threads that process virtqueue
kicks, they are bound to host CPUs/nodes.
3. Linux kernel has multiqueue block I/O.

Jens: when experimenting with multiqueue virtio-blk, how far did you
modify QEMU to eliminate global request processing state from block.c?

Stefan

Re: [Qemu-devel] Linux multiqueue block layer thoughts

[Jens Axboe]

On Wed, Nov 27 2013, Stefan Hajnoczi wrote:
> I finally got around to reading the Linux multiqueue block layer paper
> and wanted to share some thoughts about how it relates to QEMU and
> dataplane/QContext:
> http://kernel.dk/blk-mq.pdf
> 
> I think Jens has virtio-blk multiqueue patches.  So let's imagine that
> the virtio-blk device has multiple virtqueues.  (virtio-scsi is
> already multiqueue BTW.)
> 
> The paper focusses on two queue mappings: 1 queue per core and 1 queue
> per node.  In both cases the idea is to keep the block I/O code path
> localized.  This makes block I/O scale as the number of CPUs
> increases.
> 
> In QEMU we'd want to set up a mapping for the virtio-blk mq device:
> each guest vcpu or guest node has a virtio-blk virtqueue which is
> serviced by a dataplane/QContext thread.
> 
> QEMU would then process requests across these queues in parallel,
> although currently BlockDriverState is not thread-safe.  At least for
> raw we should be able to submit requests in parallel from QEMU.
> 
> Unfortunately there are some complications in the QEMU block layer:
> QEMU's own accounting, request tracking, and throttling features are
> global.  We'd need to eventually do something similar to the
> multiqueue block layer changes in the kernel to detangle this state.
> 
> Doing multiqueue for image formats is much more challenging - we'd
> have to tackle thread-safety in qcow2 and friends.  For network block
> drivers like Gluster or NBD it's also not 100% clear what the best
> approach is.  But I think the target here is local SSDs that are
> capable of high IOPs together with an SMP guest.
> 
> At the end of all this we'd arrive at the following architecture:
> 1. Guest virtio device has multiple queues (1 per node or vcpu).
> 2. QEMU has multiple dataplane/QContext threads that process virtqueue
> kicks, they are bound to host CPUs/nodes.
> 3. Linux kernel has multiqueue block I/O.

I think that sounds very reasonable. Let me know if there's anything you
need help or advice with.

> Jens: when experimenting with multiqueue virtio-blk, how far did you
> modify QEMU to eliminate global request processing state from block.c?

I did very little scaling testing on virtio-blk, it was more a demo case
for conversion than anything else. So probably not of much use to what
you are looking for...

-- 
Jens Axboe

Re: [Qemu-devel] Linux multiqueue block layer thoughts

[Stefan Hajnoczi]

On Wed, Nov 27, 2013 at 07:15:13PM -0700, Jens Axboe wrote:
> On Wed, Nov 27 2013, Stefan Hajnoczi wrote:
> > At the end of all this we'd arrive at the following architecture:
> > 1. Guest virtio device has multiple queues (1 per node or vcpu).
> > 2. QEMU has multiple dataplane/QContext threads that process virtqueue
> > kicks, they are bound to host CPUs/nodes.
> > 3. Linux kernel has multiqueue block I/O.
> 
> I think that sounds very reasonable. Let me know if there's anything you
> need help or advice with.
> 
> > Jens: when experimenting with multiqueue virtio-blk, how far did you
> > modify QEMU to eliminate global request processing state from block.c?
> 
> I did very little scaling testing on virtio-blk, it was more a demo case
> for conversion than anything else. So probably not of much use to what
> you are looking for...

Okay, thanks.  It will be a while before the whole stack supports
multiqueue but it's good to know this approach sounds reasonable.

Stefan