[Qemu-devel] questions about the bdrv_co_do_readv()

[Qemu-devel] questions about the bdrv_co_do_readv()

[Yaodong Yang]

In the block-migration period, each chunk of data is read by bdrv_aio_readv(), then bdrv_co_aio_rw_vector() , then bdrv_co_do_rw(). 

Inside the bdrv_co_do_rw() function, the bdrv_co_do_readv() function is called twice. and exit without finish the execution of the bdrv_co_do_readv() function. All these calls are inside the migration thread. Later, the iothread will finish the bottom half of the bdrv_co_do_readv() function. I do not why it is executed in this way.

In sum, the bdrv_co_do_readv() seems to be executed inside two thread, the migration thread and native iothread. Both of them executed the function twice for a single request. Could someone explain it for me ? I appreciate it very much!

Thanks!
Yaodong

Re: [Qemu-devel] questions about the bdrv_co_do_readv()

[Stefan Hajnoczi]

On Mon, Aug 26, 2013 at 09:59:51PM -0500, Yaodong Yang wrote:
> In sum, the bdrv_co_do_readv() seems to be executed inside two thread, the migration thread and native iothread. Both of them executed the function twice for a single request. Could someone explain it for me ? I appreciate it very much!

Did you check the bdrv_co_readv(bs=...) argument?  The calls may be
referring to two different BlockDriverState instances.

A raw image file looks like this:

 BDS#1 (block/raw.c)
   ->backing_hd = NULL
   ->file = BDS#2

 BDS#2 (block/raw-posix.c)

With qcow2 and no backing file you would have something like this:

 BDS#1 (block/qcow2.c)
   ->backing_hd = NULL
   ->file = BDS#2

 BDS#2 (block/raw-posix.c)

Expect to see calls forwarded from BDS#1 to BDS#2.

Stefan

Re: [Qemu-devel] questions about the bdrv_co_do_readv()

[Yaodong Yang]

[-- Attachment #1: Type: text/plain, Size: 3266 bytes --]

I checked the argument, it looks like the same. 

I guess my confusion is the follows:

code version: qemu-1.5.1

step 1:
 file:  block-migration.c, 
 function: static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)

     Inside this function, qemu calls the following function to read a chunk of data for migration.
blk->aiocb = bdrv_aio_readv(bs, cur_sector, &blk->qiov,
                                nr_sectors, blk_mig_read_cb, blk);
my questions:
No


step 2:
file block.c
function: BlockDriverAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                                 QEMUIOVector *qiov, int nb_sectors,
                                 BlockDriverCompletionFunc *cb, void *opaque)
my questions:
No

Step 3:
file block.c
function：
static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                               int64_t sector_num,
                                               QEMUIOVector *qiov,
                                               int nb_sectors,
                                               BlockDriverCompletionFunc *cb,
                                               void *opaque,
                                               bool is_write)
This function create a coroutine and then enter this coroutine. 
    co = qemu_coroutine_create(bdrv_co_do_rw);
    qemu_coroutine_enter(co, acb);

My questions:
1. What is the purpose of creating a coroutine here? 



Step 4:
file: block.c
function: static void coroutine_fn bdrv_co_do_rw(void *opaque)
In this function, the bdrv_co_do_readv() is called. and After it, it create a bottom-half and then schedule this buttom half.
acb->req.error = bdrv_co_do_readv(bs, acb->req.sector,
            acb->req.nb_sectors, acb->req.qiov, 0);

My questions:
1. is the bdrv_co_do_readv function called within the IOthread or Migration thread?

Step 5
file: block.c
function: static void bdrv_co_em_bh(void *opaque)

My questions:
1. Is this function called within the IOthread or Migration thread?

In sum,
I do not know clearly how the coroutine, bottom-half schemes are used for the migration job. is there any reference material related?

Thanks a lot !
Yaodong 








it send s the bdrv_co_do_readv(block.c) function is exectued 


On Aug 27, 2013, at 2:45 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Mon, Aug 26, 2013 at 09:59:51PM -0500, Yaodong Yang wrote:
>> In sum, the bdrv_co_do_readv() seems to be executed inside two thread, the migration thread and native iothread. Both of them executed the function twice for a single request. Could someone explain it for me ? I appreciate it very much!
> 
> Did you check the bdrv_co_readv(bs=...) argument?  The calls may be
> referring to two different BlockDriverState instances.
> 
> A raw image file looks like this:
> 
> BDS#1 (block/raw.c)
>   ->backing_hd = NULL
>   ->file = BDS#2
> 
> BDS#2 (block/raw-posix.c)
> 
> With qcow2 and no backing file you would have something like this:
> 
> BDS#1 (block/qcow2.c)
>   ->backing_hd = NULL
>   ->file = BDS#2
> 
> BDS#2 (block/raw-posix.c)
> 
> Expect to see calls forwarded from BDS#1 to BDS#2.
> 
> Stefan


[-- Attachment #2: Type: text/html, Size: 18130 bytes --]

Re: [Qemu-devel] questions about the bdrv_co_do_readv()

[Stefan Hajnoczi]

On Tue, Aug 27, 2013 at 05:55:26PM -0500, Yaodong Yang wrote:
> Step 3:
> file block.c
> function：
> static BlockDriverAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
>                                                int64_t sector_num,
>                                                QEMUIOVector *qiov,
>                                                int nb_sectors,
>                                                BlockDriverCompletionFunc *cb,
>                                                void *opaque,
>                                                bool is_write)
> This function create a coroutine and then enter this coroutine. 
>     co = qemu_coroutine_create(bdrv_co_do_rw);
>     qemu_coroutine_enter(co, acb);
> 
> My questions:
> 1. What is the purpose of creating a coroutine here? 

A coroutine can yield back to the event loop and can be resumed again
later.  This is useful for code paths where requests must wait for each
other (e.g. I/O throttling, copy-on-read, allocating writes to the same
region).

> Step 4:
> file: block.c
> function: static void coroutine_fn bdrv_co_do_rw(void *opaque)
> In this function, the bdrv_co_do_readv() is called. and After it, it create a bottom-half and then schedule this buttom half.
> acb->req.error = bdrv_co_do_readv(bs, acb->req.sector,
>             acb->req.nb_sectors, acb->req.qiov, 0);
> 
> My questions:
> 1. is the bdrv_co_do_readv function called within the IOthread or Migration thread?

It will be called in the migration thread but the global mutex has been
acquired, so the iothread must wait.

> Step 5
> file: block.c
> function: static void bdrv_co_em_bh(void *opaque)
> 
> My questions:
> 1. Is this function called within the IOthread or Migration thread?

It depends:

aio_poll() dispatches BHs that are ready, so BHs can be invoked from the
migration thread if something is calling aio_poll() (like
bdrv_read()/bdrv_pread()/etc).

In the case of bdrv_aio_readv() the BH will probably be executed later
in the iothread since the caller did not wait for the request to
complete by calling aio_poll() in a loop.

> In sum,
> I do not know clearly how the coroutine, bottom-half schemes are used for the migration job. is there any reference material related?

The migration thread grabs the global mutex before accessing the
BlockDriverState, so the iothread has to wait.  Therefore you don't
really need to worry about which thread you are in.

I hope my answers have made the relationships a bit clearer for you.
Ultimately you need to read the qemu.git/master code, there is no stable
API and things change over time.

Stefan