[Qemu-devel] Block layer roadmap

[Qemu-devel] Block layer roadmap

[Stefan Hajnoczi]

Hi,
Here is a list of block layer and storage changes that have been discussed.  It
is useful to have a roadmap of changes in order to avoid duplication, allow
more developers to contribute, and to communicate the direction of storage in
QEMU.

I suggest we first do a braindump of all changes that have been discussed.
Later we can discuss specific changes and if/when they fit into the roadmap -
please don't jump into discussion about specific changes yet.

Kevin: I hope this a useful starting point.  Here are all the items
that I am aware of:

=Material for next QEMU release=

Coroutines in the block layer [Kevin]
 * Programming model to simplify block drivers without blocking QEMU threads

Generic copy-on-read [Stefan]
 * Populate image file to avoid fetching same block from backing file
again later

Generic image streaming [Stefan]
 * Make block_stream commands available for all image formats that
support backing files

Live block copy [Marcelo/Kevin/Stefan?]
 * Copy the contents of an image file while a guest is using it

In-place qcow2 <-> qed conversion [Devin, GSoC 2011]:
 * Fast conversion between qcow2 and qed image formats without copy all data

VMDK enhancements [Fam, GSoC 2011]
 * Implement latest VMDK specs to support modern image files

Block I/O limits [Zhi Yong]
 * Resource control for guest I/O bandwidth/iops consumption

=Changes where I am not aware of firm plans=

Cow overlay [Dong Xu "Robert"]
 * Allow live block copy and image streaming to raw destination files

snapshot_blkdev and Backup API [Jes, Jagane]
 * Support for consistent disk snapshots and dirty block tracking
 * Allow backup software to integrate with QEMU

-blockdev [Markus?]
 * Explicit user control over block device trees

QCOW3
 * Extend qcow2 format to address current and future image format challenges

iSCSI/NBD/Remote block device integration
 * Enable remote access to disk images for live migration and other tasks

Pre/post block copy
 * Working block migration

Avoid blocking QEMU threads
 * Today loss of NFS connectivity can hang guests
 * It's critical never to block the vcpu thread
 * The iothread should also not block while the qemu mutex is held
 * All blocking operations must be done asynchronously or in a worker thread

virtio-scsi [Paolo/Stefan]
 * The next step after virtio-blk, full SCSI command set and appears
as SCSI HBA in guest

tcm_vhost [Stefan]
 * Directly connect virtio-scsi with Linux in-kernel SCSI target
 * Pass-through of host SCSI devices

Re: [Qemu-devel] Block layer roadmap

[Anthony Liguori]

On 07/27/2011 07:37 AM, Stefan Hajnoczi wrote:
> Hi,
> Here is a list of block layer and storage changes that have been discussed.  It
> is useful to have a roadmap of changes in order to avoid duplication, allow
> more developers to contribute, and to communicate the direction of storage in
> QEMU.

Thanks for writing this up Stefan!

> I suggest we first do a braindump of all changes that have been discussed.
> Later we can discuss specific changes and if/when they fit into the roadmap -
> please don't jump into discussion about specific changes yet.
>
> Kevin: I hope this a useful starting point.  Here are all the items
> that I am aware of:
>
> =Material for next QEMU release=
>
> Coroutines in the block layer [Kevin]
>   * Programming model to simplify block drivers without blocking QEMU threads
>
> Generic copy-on-read [Stefan]
>   * Populate image file to avoid fetching same block from backing file
> again later
>
> Generic image streaming [Stefan]
>   * Make block_stream commands available for all image formats that
> support backing files
>
> Live block copy [Marcelo/Kevin/Stefan?]
>   * Copy the contents of an image file while a guest is using it
>
> In-place qcow2<->  qed conversion [Devin, GSoC 2011]:
>   * Fast conversion between qcow2 and qed image formats without copy all data
>
> VMDK enhancements [Fam, GSoC 2011]
>   * Implement latest VMDK specs to support modern image files
>
> Block I/O limits [Zhi Yong]
>   * Resource control for guest I/O bandwidth/iops consumption
>
> =Changes where I am not aware of firm plans=
>
> Cow overlay [Dong Xu "Robert"]
>   * Allow live block copy and image streaming to raw destination files
>
> snapshot_blkdev and Backup API [Jes, Jagane]
>   * Support for consistent disk snapshots and dirty block tracking
>   * Allow backup software to integrate with QEMU
>
> -blockdev [Markus?]
>   * Explicit user control over block device trees

I'm planning on helping out here however I can.

Regards,

Anthony Liguori

> QCOW3
>   * Extend qcow2 format to address current and future image format challenges
>
> iSCSI/NBD/Remote block device integration
>   * Enable remote access to disk images for live migration and other tasks
>
> Pre/post block copy
>   * Working block migration
>
> Avoid blocking QEMU threads
>   * Today loss of NFS connectivity can hang guests
>   * It's critical never to block the vcpu thread
>   * The iothread should also not block while the qemu mutex is held
>   * All blocking operations must be done asynchronously or in a worker thread
>
> virtio-scsi [Paolo/Stefan]
>   * The next step after virtio-blk, full SCSI command set and appears
> as SCSI HBA in guest
>
> tcm_vhost [Stefan]
>   * Directly connect virtio-scsi with Linux in-kernel SCSI target
>   * Pass-through of host SCSI devices
>

Re: [Qemu-devel] Block layer roadmap

[Kevin Wolf]

Am 27.07.2011 14:37, schrieb Stefan Hajnoczi:
> Hi,
> Here is a list of block layer and storage changes that have been discussed.  It
> is useful to have a roadmap of changes in order to avoid duplication, allow
> more developers to contribute, and to communicate the direction of storage in
> QEMU.
> 
> I suggest we first do a braindump of all changes that have been discussed.
> Later we can discuss specific changes and if/when they fit into the roadmap -
> please don't jump into discussion about specific changes yet.
> 
> Kevin: I hope this a useful starting point.  Here are all the items
> that I am aware of:
> 
> =Material for next QEMU release=
> 
> Coroutines in the block layer [Kevin]
>  * Programming model to simplify block drivers without blocking QEMU threads
> 
> Generic copy-on-read [Stefan]
>  * Populate image file to avoid fetching same block from backing file
> again later
> 
> Generic image streaming [Stefan]
>  * Make block_stream commands available for all image formats that
> support backing files
> 
> Live block copy [Marcelo/Kevin/Stefan?]
>  * Copy the contents of an image file while a guest is using it
> 
> In-place qcow2 <-> qed conversion [Devin, GSoC 2011]:
>  * Fast conversion between qcow2 and qed image formats without copy all data
> 
> VMDK enhancements [Fam, GSoC 2011]
>  * Implement latest VMDK specs to support modern image files
> 
> Block I/O limits [Zhi Yong]
>  * Resource control for guest I/O bandwidth/iops consumption

Another item that just came up on IRC again: Allow guests to toggle WCE,
so that we finally can get rid of the cache=writethrough default. We
should definitely get this done before 1.0.

Kevin

Re: [Qemu-devel] Block layer roadmap

[Christoph Hellwig]

On Thu, Jul 28, 2011 at 12:05:43PM +0200, Kevin Wolf wrote:
> Another item that just came up on IRC again: Allow guests to toggle WCE,
> so that we finally can get rid of the cache=writethrough default. We
> should definitely get this done before 1.0.

Guest toggling is nice and cool, but the real feature is to allow it on
the command line.  As mentioned about 10 times before guests generally don't
fiddle with it without the administrator triggering it manually.  So
specifying it in the qemu config is the much better interface for 95% of
the use cases.

Re: [Qemu-devel] Block layer roadmap

[Kevin Wolf]

Am 28.07.2011 14:08, schrieb Christoph Hellwig:
> On Thu, Jul 28, 2011 at 12:05:43PM +0200, Kevin Wolf wrote:
>> Another item that just came up on IRC again: Allow guests to toggle WCE,
>> so that we finally can get rid of the cache=writethrough default. We
>> should definitely get this done before 1.0.
> 
> Guest toggling is nice and cool, but the real feature is to allow it on
> the command line.  As mentioned about 10 times before guests generally don't
> fiddle with it without the administrator triggering it manually.  So
> specifying it in the qemu config is the much better interface for 95% of
> the use cases.

I don't really care about guest toggling. I care about the default cache
mode, and Anthony said that he doesn't agree with changing it unless
guests can toggle it.

Kevin

Re: [Qemu-devel] Block layer roadmap

[Stefan Hajnoczi]

On Wed, Jul 27, 2011 at 1:37 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> =Changes where I am not aware of firm plans=

qcow2 online resize
 * Handle snapshots
 * Support shrinking

qed online resize
 * Support shrinking

Re: [Qemu-devel] Block layer roadmap

[Christoph Hellwig]

On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
> Coroutines in the block layer [Kevin]
>  * Programming model to simplify block drivers without blocking QEMU threads

Can anyone explain what the whole point of this is?  It really just is
a bit of syntactic sugar for the current async state machines.  What does
it buy us over going for real threading?

Re: [Qemu-devel] Block layer roadmap

[Frediano Ziglio]

2011/7/28 Christoph Hellwig <hch@lst.de>:
> On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
>> Coroutines in the block layer [Kevin]
>>  * Programming model to simplify block drivers without blocking QEMU threads
>
> Can anyone explain what the whole point of this is?  It really just is
> a bit of syntactic sugar for the current async state machines.  What does
> it buy us over going for real threading?
>

This has nothing (or few) to do with threads. Instead of splitting
functions in callbacks at every synchronous function it allow to write
more readable code. For instance after changing qcow code from current
to coroutine you remove about 400 lines (about 30%). This will help
maintaining code and develop more complicated optimizations.

About threading you can do threading using AIO and using coroutines.

Frediano

Re: [Qemu-devel] Block layer roadmap

[Christoph Hellwig]

On Thu, Jul 28, 2011 at 02:15:38PM +0200, Frediano Ziglio wrote:
> 
> This has nothing (or few) to do with threads. Instead of splitting
> functions in callbacks at every synchronous function it allow to write
> more readable code.

Thanks for repeating my statement that it doesn't fix the real thing
but just adds syntactic sugar.  

Re: [Qemu-devel] Block layer roadmap

[Kevin Wolf]

Am 28.07.2011 14:09, schrieb Christoph Hellwig:
> On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
>> Coroutines in the block layer [Kevin]
>>  * Programming model to simplify block drivers without blocking QEMU threads
> 
> Can anyone explain what the whole point of this is?  It really just is
> a bit of syntactic sugar for the current async state machines.  What does
> it buy us over going for real threading?

The only current block driver that really does everything in an async
state machine is qed. It's definitely not nice code, and having to
convert all of the other block drivers to this would be a lot of work.

So if it only means that we're making things async that would block the
VCPU until now, isn't that a great improvement already?

The advantage compared to threading is that it allows an easy and
incremental transition.

Kevin

Re: [Qemu-devel] Block layer roadmap

[Stefan Hajnoczi]

On Thu, Jul 28, 2011 at 1:35 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 28.07.2011 14:09, schrieb Christoph Hellwig:
>> On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
>>> Coroutines in the block layer [Kevin]
>>>  * Programming model to simplify block drivers without blocking QEMU threads
>>
>> Can anyone explain what the whole point of this is?  It really just is
>> a bit of syntactic sugar for the current async state machines.  What does
>> it buy us over going for real threading?
>
> The only current block driver that really does everything in an async
> state machine is qed. It's definitely not nice code, and having to
> convert all of the other block drivers to this would be a lot of work.

Thanks Kevin :).  I do agree with the clumsiness of async callback
programming - a lot of code is spent bundling and unbundling variables
to pass between functions, not to mention that the control flow is
much harder to follow.

> So if it only means that we're making things async that would block the
> VCPU until now, isn't that a great improvement already?
>
> The advantage compared to threading is that it allows an easy and
> incremental transition.

Also remember that we already have a threads-based implementation of
coroutines.  Later on we can do fine-grained locking and switch to
threads directly instead of coroutines, if need be.

Stefan

Re: [Qemu-devel] Block layer roadmap

[Kevin Wolf]

Am 28.07.2011 14:54, schrieb Stefan Hajnoczi:
> On Thu, Jul 28, 2011 at 1:35 PM, Kevin Wolf <kwolf@redhat.com> wrote:
>> Am 28.07.2011 14:09, schrieb Christoph Hellwig:
>>> On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
>>>> Coroutines in the block layer [Kevin]
>>>>  * Programming model to simplify block drivers without blocking QEMU threads
>>>
>>> Can anyone explain what the whole point of this is?  It really just is
>>> a bit of syntactic sugar for the current async state machines.  What does
>>> it buy us over going for real threading?
>>
>> The only current block driver that really does everything in an async
>> state machine is qed. It's definitely not nice code, and having to
>> convert all of the other block drivers to this would be a lot of work.
> 
> Thanks Kevin :). 

I certainly didn't mean to attack your code or even yourself. It's not
that qed is done particularly bad or anything. That the code isn't
really nice is just the natural result of the callback-based programming
model.

>> So if it only means that we're making things async that would block the
>> VCPU until now, isn't that a great improvement already?
>>
>> The advantage compared to threading is that it allows an easy and
>> incremental transition.
> 
> Also remember that we already have a threads-based implementation of
> coroutines.  Later on we can do fine-grained locking and switch to
> threads directly instead of coroutines, if need be.

Might be an option for the future, but for now there's enough left to do
to take real advantage of coroutines.

Kevin

Re: [Qemu-devel] Block layer roadmap

[Stefan Hajnoczi]

On Thu, Jul 28, 2011 at 2:10 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 28.07.2011 14:54, schrieb Stefan Hajnoczi:
>> On Thu, Jul 28, 2011 at 1:35 PM, Kevin Wolf <kwolf@redhat.com> wrote:
>>> Am 28.07.2011 14:09, schrieb Christoph Hellwig:
>>>> On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
>>>>> Coroutines in the block layer [Kevin]
>>>>>  * Programming model to simplify block drivers without blocking QEMU threads
>>>>
>>>> Can anyone explain what the whole point of this is?  It really just is
>>>> a bit of syntactic sugar for the current async state machines.  What does
>>>> it buy us over going for real threading?
>>>
>>> The only current block driver that really does everything in an async
>>> state machine is qed. It's definitely not nice code, and having to
>>> convert all of the other block drivers to this would be a lot of work.
>>
>> Thanks Kevin :).
>
> I certainly didn't mean to attack your code or even yourself. It's not
> that qed is done particularly bad or anything. That the code isn't
> really nice is just the natural result of the callback-based programming
> model.

No worries, no offence taken :)

Stefan

Re: [Qemu-devel] Block layer roadmap

[Anthony Liguori]

On 07/28/2011 07:09 AM, Christoph Hellwig wrote:
> On Wed, Jul 27, 2011 at 01:37:31PM +0100, Stefan Hajnoczi wrote:
>> Coroutines in the block layer [Kevin]
>>   * Programming model to simplify block drivers without blocking QEMU threads
>
> Can anyone explain what the whole point of this is?  It really just is
> a bit of syntactic sugar for the current async state machines.  What does
> it buy us over going for real threading?

It is threading--just with a common locking model where a single big 
lock is held to make up for the fact that most of QEMU isn't reentrant.

By restructuring the code to be threaded, we can incrementally remove 
the big lock if we audit for use of non-reentrant functions and 
introduce more granular locking.

The whole ucontext/setjmp thing is just an optimization.  I would hope 
it entirely disappears long term as we promote coroutines to full threads.

Regards,

Anthony Liguori

>
>

Re: [Qemu-devel] Block layer roadmap

[Stefan Hajnoczi]

Frediano suggested:

Enhanced volume key in qcow3
 * luks-like key scheme that allows changing passphrase without
re-encrypting image data