Re: [Qemu-devel] [PATCH 2/6] sheepdog: restart I/O when socket becomes ready in do_co_req()

Re: [Qemu-devel] [PATCH 2/6] sheepdog: restart I/O when socket becomes ready in do_co_req()

[Kevin Wolf]

Am 27.06.2012 00:26, schrieb MORITA Kazutaka:
> Currently, no one reenters the yielded coroutine.  This fixes it.
> 
> Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
> ---
>  block/sheepdog.c |   14 ++++++++++++++
>  1 files changed, 14 insertions(+), 0 deletions(-)

Paolo, is this how qemu_co_recv/send are supposed to be used? Shouldn't
the functions take care of reentering the coroutine like the block
functions do?

Kevin

> 
> diff --git a/block/sheepdog.c b/block/sheepdog.c
> index afd06aa..0b49c6d 100644
> --- a/block/sheepdog.c
> +++ b/block/sheepdog.c
> @@ -577,10 +577,21 @@ out:
>      return ret;
>  }
>  
> +static void restart_co_req(void *opaque)
> +{
> +    Coroutine *co = opaque;
> +
> +    qemu_coroutine_enter(co, NULL);
> +}
> +
>  static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
>                                    unsigned int *wlen, unsigned int *rlen)
>  {
>      int ret;
> +    Coroutine *co;
> +
> +    co = qemu_coroutine_self();
> +    qemu_aio_set_fd_handler(sockfd, NULL, restart_co_req, NULL, co);
>  
>      socket_set_block(sockfd);
>      ret = send_co_req(sockfd, hdr, data, wlen);
> @@ -588,6 +599,8 @@ static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
>          goto out;
>      }
>  
> +    qemu_aio_set_fd_handler(sockfd, restart_co_req, NULL, NULL, co);
> +
>      ret = qemu_co_recv(sockfd, hdr, sizeof(*hdr));
>      if (ret < sizeof(*hdr)) {
>          error_report("failed to get a rsp, %s", strerror(errno));
> @@ -609,6 +622,7 @@ static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
>      }
>      ret = 0;
>  out:
> +    qemu_aio_set_fd_handler(sockfd, NULL, NULL, NULL, NULL);
>      socket_set_nonblock(sockfd);
>      return ret;
>  }
> 

Re: [Qemu-devel] [PATCH 2/6] sheepdog: restart I/O when socket becomes ready in do_co_req()

[Paolo Bonzini]

Il 03/07/2012 15:09, Kevin Wolf ha scritto:
>> > Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
>> > ---
>> >  block/sheepdog.c |   14 ++++++++++++++
>> >  1 files changed, 14 insertions(+), 0 deletions(-)
> Paolo, is this how qemu_co_recv/send are supposed to be used?

Yes.

> Shouldn't
> the functions take care of reentering the coroutine like the block
> functions do?

No, because qemu_co_recv/send are generic outside the block layer and
they don't have enough context from the caller to set up the handler.
For example qemu_co_send to keep the recv handler, but in turn the recv
handler might have its own opaque.

In fact, qemu_co_recv/send originated in sheepdog :) so they are (even
with all the refactoring that went in since) Kazutaka's pet too!

Paolo

Re: [Qemu-devel] [PATCH 3/6] sheepdog: use coroutine based socket functions in coroutine context

[Kevin Wolf]

Am 27.06.2012 00:26, schrieb MORITA Kazutaka:
> This removes blocking network I/Os in coroutine context.
> 
> Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
> ---
>  block/sheepdog.c |   10 ++++++++--
>  1 files changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/block/sheepdog.c b/block/sheepdog.c
> index 0b49c6d..5dc1d7a 100644
> --- a/block/sheepdog.c
> +++ b/block/sheepdog.c
> @@ -541,11 +541,18 @@ static coroutine_fn int send_co_req(int sockfd, SheepdogReq *hdr, void *data,
>      return ret;
>  }
>  
> +static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
> +                                  unsigned int *wlen, unsigned int *rlen);
> +
>  static int do_req(int sockfd, SheepdogReq *hdr, void *data,
>                    unsigned int *wlen, unsigned int *rlen)
>  {
>      int ret;
>  
> +    if (qemu_in_coroutine()) {
> +        return do_co_req(sockfd, hdr, data, wlen, rlen);
> +    }
> +
>      socket_set_block(sockfd);
>      ret = send_req(sockfd, hdr, data, wlen);
>      if (ret < 0) {

How about replacing the non-coroutine implementation by code that
creates a new coroutine and executes do_co_req() as well? This would
reduce some code duplication.

Kevin

Re: [Qemu-devel] [PATCH 3/6] sheepdog: use coroutine based socket functions in coroutine context

[Peter Crosthwaite]

On Tue, Jul 3, 2012 at 11:15 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 27.06.2012 00:26, schrieb MORITA Kazutaka:
>> This removes blocking network I/Os in coroutine context.
>>
>> Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
>> ---
>>  block/sheepdog.c |   10 ++++++++--
>>  1 files changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/block/sheepdog.c b/block/sheepdog.c
>> index 0b49c6d..5dc1d7a 100644
>> --- a/block/sheepdog.c
>> +++ b/block/sheepdog.c
>> @@ -541,11 +541,18 @@ static coroutine_fn int send_co_req(int sockfd, SheepdogReq *hdr, void *data,
>>      return ret;
>>  }
>>
>> +static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
>> +                                  unsigned int *wlen, unsigned int *rlen);
>> +
>>  static int do_req(int sockfd, SheepdogReq *hdr, void *data,
>>                    unsigned int *wlen, unsigned int *rlen)
>>  {
>>      int ret;
>>
>> +    if (qemu_in_coroutine()) {
>> +        return do_co_req(sockfd, hdr, data, wlen, rlen);
>> +    }
>> +
>>      socket_set_block(sockfd);
>>      ret = send_req(sockfd, hdr, data, wlen);
>>      if (ret < 0) {
>
> How about replacing the non-coroutine implementation by code that
> creates a new coroutine and executes do_co_req() as well? This would
> reduce some code duplication.
>

+1. I presume it can it be done such that there is no if
(qemu_in_coroutine()) logic that way?

Regards,
Peter

> Kevin
>

Re: [Qemu-devel] [PATCH 3/6] sheepdog: use coroutine based socket functions in coroutine context

[Kevin Wolf]

Am 04.07.2012 02:25, schrieb Peter Crosthwaite:
> On Tue, Jul 3, 2012 at 11:15 PM, Kevin Wolf <kwolf@redhat.com> wrote:
>> Am 27.06.2012 00:26, schrieb MORITA Kazutaka:
>>> This removes blocking network I/Os in coroutine context.
>>>
>>> Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
>>> ---
>>>  block/sheepdog.c |   10 ++++++++--
>>>  1 files changed, 8 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/block/sheepdog.c b/block/sheepdog.c
>>> index 0b49c6d..5dc1d7a 100644
>>> --- a/block/sheepdog.c
>>> +++ b/block/sheepdog.c
>>> @@ -541,11 +541,18 @@ static coroutine_fn int send_co_req(int sockfd, SheepdogReq *hdr, void *data,
>>>      return ret;
>>>  }
>>>
>>> +static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
>>> +                                  unsigned int *wlen, unsigned int *rlen);
>>> +
>>>  static int do_req(int sockfd, SheepdogReq *hdr, void *data,
>>>                    unsigned int *wlen, unsigned int *rlen)
>>>  {
>>>      int ret;
>>>
>>> +    if (qemu_in_coroutine()) {
>>> +        return do_co_req(sockfd, hdr, data, wlen, rlen);
>>> +    }
>>> +
>>>      socket_set_block(sockfd);
>>>      ret = send_req(sockfd, hdr, data, wlen);
>>>      if (ret < 0) {
>>
>> How about replacing the non-coroutine implementation by code that
>> creates a new coroutine and executes do_co_req() as well? This would
>> reduce some code duplication.
>>
> 
> +1. I presume it can it be done such that there is no if
> (qemu_in_coroutine()) logic that way?

That was not my intention, and actually I don't think it would help your
case here because this is a function truly internal to the block layer
(or actually even a single block driver).

Kevin

Re: [Qemu-devel] [PATCH 3/6] sheepdog: use coroutine based socket functions in coroutine context

[MORITA Kazutaka]

At Tue, 03 Jul 2012 15:15:03 +0200,
Kevin Wolf wrote:
> 
> Am 27.06.2012 00:26, schrieb MORITA Kazutaka:
> > This removes blocking network I/Os in coroutine context.
> > 
> > Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
> > ---
> >  block/sheepdog.c |   10 ++++++++--
> >  1 files changed, 8 insertions(+), 2 deletions(-)
> > 
> > diff --git a/block/sheepdog.c b/block/sheepdog.c
> > index 0b49c6d..5dc1d7a 100644
> > --- a/block/sheepdog.c
> > +++ b/block/sheepdog.c
> > @@ -541,11 +541,18 @@ static coroutine_fn int send_co_req(int sockfd, SheepdogReq *hdr, void *data,
> >      return ret;
> >  }
> >  
> > +static coroutine_fn int do_co_req(int sockfd, SheepdogReq *hdr, void *data,
> > +                                  unsigned int *wlen, unsigned int *rlen);
> > +
> >  static int do_req(int sockfd, SheepdogReq *hdr, void *data,
> >                    unsigned int *wlen, unsigned int *rlen)
> >  {
> >      int ret;
> >  
> > +    if (qemu_in_coroutine()) {
> > +        return do_co_req(sockfd, hdr, data, wlen, rlen);
> > +    }
> > +
> >      socket_set_block(sockfd);
> >      ret = send_req(sockfd, hdr, data, wlen);
> >      if (ret < 0) {
> 
> How about replacing the non-coroutine implementation by code that
> creates a new coroutine and executes do_co_req() as well? This would
> reduce some code duplication.

Indeed.  I'll send a patch for it soon.

Thanks,

Kazutaka

Re: [Qemu-devel] [PATCH 0/6] sheepdog: various fixes

[Kevin Wolf]

Am 27.06.2012 00:26, schrieb MORITA Kazutaka:
> See individual patches for details.
> 
> MORITA Kazutaka (6):
>   sheepdog: fix dprintf format strings
>   sheepdog: restart I/O when socket becomes ready in do_co_req()
>   sheepdog: use coroutine based socket functions in coroutine context
>   sheepdog: make sure we don't free aiocb before sending all requests
>   sheepdog: split outstanding list into inflight and pending
>   sheepdog: traverse pending_list from the first for each time
> 
>  block/sheepdog.c |  130 +++++++++++++++++++++++++++++++++--------------------
>  1 files changed, 81 insertions(+), 49 deletions(-)

Thanks, applied all to the block branch.

Kevin