Re: [Qemu-devel] Question about QEMU's threading model and stacking multiple block drivers

[Max Reitz <mreitz@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2337 bytes --]

CC-ing qemu-block, Stefan, Fam.


On 08.02.2017 03:38, Adrian Suarez wrote:
> We’ve implemented a block driver that exposes storage to QEMU VMs. Our
> block driver (O) is interposing on writes to some other type of storage
> (B). O performs low latency replication and then asynchronously issues the
> write to the backing block driver, B, using bdrv_aio_writev(). Our problem
> is that the write latencies seen by the workload in the guest should be
> those imposed by O plus the guest I/O and QEMU stack (around 25us total
> based on our measurements), but we’re actually seeing much higher latencies
> (around 120us). We suspect that this is due to the backing block driver B’s
> coroutines blocking our coroutines. The sequence of events is as follows
> (see diagram:
> https://docs.google.com/drawings/d/12h1QbecvxzlKxSFvGKYAzvAJ18kTW6AVTwDR6VA8hkw/pub?w=576&h=565
> ):
> 
> 1. Write is issued to our block driver O using the asynchronous interface
> for QEMU block driver.
> 2. Write is replicated to a fast device asynchronously.
> 2.a. In a different thread, the fast device invokes a callback on
> completion that causes a coroutine to be scheduled to run in the QEMU
> iothread that acknowledges completion of the write to the guest OS.
> 2.b. The coroutine scheduled in (2.a) is executed.
> 3. Write is issued asynchronously to the backing block driver, B.
> 3.a. The backing block driver, B, invokes the completion function supplied
> by us, which frees any memory associated with the write (e.g. copies of IO
> vectors).
> 
> Steps (1), (2), and (3) are performed in the same coroutine (our driver's
> bdrv_aio_writev() implementation). (2.a) is executed in a thread that is
> part of our transport library linked by O, and (2.b) and (3.a) are executed
> as coroutines in the QEMU iothread.
> 
> We've tried improving the performance by using separate iothreads for the
> two devices, but this only shaved about lowered the latency to around 100us
> and caused stability issues. What's the best way to create a separate
> iothread for the backing driver to do all of its work in?

I don't think it's possible to use different AioContexts for
BlockDriverStates in the same BDS chain, at least not currently. But
others may know more about this.

Max

> 
> -Adrian
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]