[Qemu-devel] [PATCH v2] virtio-blk: Fix double completion for werror=stop

[Qemu-devel] [PATCH v2] virtio-blk: Fix double completion for werror=stop

[Fam Zheng]

When a request R is absorbed by request M, it is appended to the
"mr_next" queue led by M, and is completed together with the completion
of M, in virtio_blk_rw_complete.

With error policy equals stop, if M has an I/O error, now R also gets
prepended to the per device DMA restart queue, which will be retried
when VM resumes.  It leads to a double completion (in symptoms of memory
corruption or use after free).

Adding R to the queue is superfluous, only M needs to be in the queue.

Fix this by marking request R as "merged" and skipping it in
virtio_blk_handle_rw_error.

Cc: qemu-stable@nongnu.org
Signed-off-by: Fam Zheng <famz@redhat.com>

---

v2: Don't lose the request in migration. [Paolo]
---
 hw/block/virtio-blk.c          | 7 +++++++
 include/hw/virtio/virtio-blk.h | 1 +
 2 files changed, 8 insertions(+)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index e70fccf..5cdb06f 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -36,6 +36,7 @@ VirtIOBlockReq *virtio_blk_alloc_request(VirtIOBlock *s)
     req->in_len = 0;
     req->next = NULL;
     req->mr_next = NULL;
+    req->merged = false;
     return req;
 }
 
@@ -344,6 +345,7 @@ static inline void submit_requests(BlockBackend *blk, MultiReqBuffer *mrb,
         for (i = start + 1; i < start + num_reqs; i++) {
             qemu_iovec_concat(qiov, &mrb->reqs[i]->qiov, 0,
                               mrb->reqs[i]->qiov.size);
+            mrb->reqs[i]->merged = true;
             mrb->reqs[i - 1]->mr_next = mrb->reqs[i];
             nb_sectors += mrb->reqs[i]->qiov.size / BDRV_SECTOR_SIZE;
         }
@@ -511,6 +513,11 @@ void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
               - sizeof(struct virtio_blk_inhdr);
     iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
 
+    if (req->merged) {
+        /* Enough for restarting a (migrated) merged request, no need to
+         * actually submit I/O. */
+        return;
+    }
     type = virtio_ldl_p(VIRTIO_DEVICE(req->dev), &req->out.type);
 
     /* VIRTIO_BLK_T_OUT defines the command direction. VIRTIO_BLK_T_BARRIER
diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
index 6bf5905..db4adf4 100644
--- a/include/hw/virtio/virtio-blk.h
+++ b/include/hw/virtio/virtio-blk.h
@@ -70,6 +70,7 @@ typedef struct VirtIOBlockReq {
     size_t in_len;
     struct VirtIOBlockReq *next;
     struct VirtIOBlockReq *mr_next;
+    bool merged;
     BlockAcctCookie acct;
 } VirtIOBlockReq;
 
-- 
2.4.3

Re: [Qemu-devel] [PATCH v2] virtio-blk: Fix double completion for werror=stop

[Laurent Vivier]

On 16/11/2015 07:10, Fam Zheng wrote:
> When a request R is absorbed by request M, it is appended to the
> "mr_next" queue led by M, and is completed together with the completion
> of M, in virtio_blk_rw_complete.
> 
> With error policy equals stop, if M has an I/O error, now R also gets
> prepended to the per device DMA restart queue, which will be retried
> when VM resumes.  It leads to a double completion (in symptoms of memory
> corruption or use after free).
> 
> Adding R to the queue is superfluous, only M needs to be in the queue.
> 
> Fix this by marking request R as "merged" and skipping it in
> virtio_blk_handle_rw_error.
> 
> Cc: qemu-stable@nongnu.org
> Signed-off-by: Fam Zheng <famz@redhat.com>
> 
> ---
> 
> v2: Don't lose the request in migration. [Paolo]

Tested-by: Laurent Vivier <lvivier@redhat.com>


> ---
>  hw/block/virtio-blk.c          | 7 +++++++
>  include/hw/virtio/virtio-blk.h | 1 +
>  2 files changed, 8 insertions(+)
> 
> diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> index e70fccf..5cdb06f 100644
> --- a/hw/block/virtio-blk.c
> +++ b/hw/block/virtio-blk.c
> @@ -36,6 +36,7 @@ VirtIOBlockReq *virtio_blk_alloc_request(VirtIOBlock *s)
>      req->in_len = 0;
>      req->next = NULL;
>      req->mr_next = NULL;
> +    req->merged = false;
>      return req;
>  }
>  
> @@ -344,6 +345,7 @@ static inline void submit_requests(BlockBackend *blk, MultiReqBuffer *mrb,
>          for (i = start + 1; i < start + num_reqs; i++) {
>              qemu_iovec_concat(qiov, &mrb->reqs[i]->qiov, 0,
>                                mrb->reqs[i]->qiov.size);
> +            mrb->reqs[i]->merged = true;
>              mrb->reqs[i - 1]->mr_next = mrb->reqs[i];
>              nb_sectors += mrb->reqs[i]->qiov.size / BDRV_SECTOR_SIZE;
>          }
> @@ -511,6 +513,11 @@ void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
>                - sizeof(struct virtio_blk_inhdr);
>      iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
>  
> +    if (req->merged) {
> +        /* Enough for restarting a (migrated) merged request, no need to
> +         * actually submit I/O. */
> +        return;
> +    }
>      type = virtio_ldl_p(VIRTIO_DEVICE(req->dev), &req->out.type);
>  
>      /* VIRTIO_BLK_T_OUT defines the command direction. VIRTIO_BLK_T_BARRIER
> diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
> index 6bf5905..db4adf4 100644
> --- a/include/hw/virtio/virtio-blk.h
> +++ b/include/hw/virtio/virtio-blk.h
> @@ -70,6 +70,7 @@ typedef struct VirtIOBlockReq {
>      size_t in_len;
>      struct VirtIOBlockReq *next;
>      struct VirtIOBlockReq *mr_next;
> +    bool merged;
>      BlockAcctCookie acct;
>  } VirtIOBlockReq;
>  
> 

Re: [Qemu-devel] [PATCH v2] virtio-blk: Fix double completion for werror=stop

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3221 bytes --]

On Mon, Nov 16, 2015 at 02:10:36PM +0800, Fam Zheng wrote:
> When a request R is absorbed by request M, it is appended to the
> "mr_next" queue led by M, and is completed together with the completion
> of M, in virtio_blk_rw_complete.
> 
> With error policy equals stop, if M has an I/O error, now R also gets
> prepended to the per device DMA restart queue, which will be retried
> when VM resumes.  It leads to a double completion (in symptoms of memory
> corruption or use after free).
> 
> Adding R to the queue is superfluous, only M needs to be in the queue.
> 
> Fix this by marking request R as "merged" and skipping it in
> virtio_blk_handle_rw_error.
> 
> Cc: qemu-stable@nongnu.org
> Signed-off-by: Fam Zheng <famz@redhat.com>
> 
> ---
> 
> v2: Don't lose the request in migration. [Paolo]
> ---
>  hw/block/virtio-blk.c          | 7 +++++++
>  include/hw/virtio/virtio-blk.h | 1 +
>  2 files changed, 8 insertions(+)
> 
> diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> index e70fccf..5cdb06f 100644
> --- a/hw/block/virtio-blk.c
> +++ b/hw/block/virtio-blk.c
> @@ -36,6 +36,7 @@ VirtIOBlockReq *virtio_blk_alloc_request(VirtIOBlock *s)
>      req->in_len = 0;
>      req->next = NULL;
>      req->mr_next = NULL;
> +    req->merged = false;
>      return req;
>  }
>  
> @@ -344,6 +345,7 @@ static inline void submit_requests(BlockBackend *blk, MultiReqBuffer *mrb,
>          for (i = start + 1; i < start + num_reqs; i++) {
>              qemu_iovec_concat(qiov, &mrb->reqs[i]->qiov, 0,
>                                mrb->reqs[i]->qiov.size);
> +            mrb->reqs[i]->merged = true;
>              mrb->reqs[i - 1]->mr_next = mrb->reqs[i];
>              nb_sectors += mrb->reqs[i]->qiov.size / BDRV_SECTOR_SIZE;
>          }
> @@ -511,6 +513,11 @@ void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
>                - sizeof(struct virtio_blk_inhdr);
>      iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
>  
> +    if (req->merged) {
> +        /* Enough for restarting a (migrated) merged request, no need to
> +         * actually submit I/O. */
> +        return;
> +    }
>      type = virtio_ldl_p(VIRTIO_DEVICE(req->dev), &req->out.type);
>  
>      /* VIRTIO_BLK_T_OUT defines the command direction. VIRTIO_BLK_T_BARRIER
> diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
> index 6bf5905..db4adf4 100644
> --- a/include/hw/virtio/virtio-blk.h
> +++ b/include/hw/virtio/virtio-blk.h
> @@ -70,6 +70,7 @@ typedef struct VirtIOBlockReq {
>      size_t in_len;
>      struct VirtIOBlockReq *next;
>      struct VirtIOBlockReq *mr_next;
> +    bool merged;
>      BlockAcctCookie acct;
>  } VirtIOBlockReq;

I'm not sure if this patch truly fixes the bug:

virtio_blk_rw_complete() doesn't do req->mr_next = NULL.  There is a
potential double-free if resubmitting failed requests doesn't overwrite
themr_next field.

This can be fixed by adding req->mr_next = NULL to the loop in
virtio_blk_rw_complete().

Is that enough to solve the bug?  I don't think adding a new field is
necessary.

If not, please explain the double-free.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v2] virtio-blk: Fix double completion for werror=stop

[Fam Zheng]

On Tue, 11/17 14:58, Stefan Hajnoczi wrote:
> On Mon, Nov 16, 2015 at 02:10:36PM +0800, Fam Zheng wrote:
> > When a request R is absorbed by request M, it is appended to the
> > "mr_next" queue led by M, and is completed together with the completion
> > of M, in virtio_blk_rw_complete.
> > 
> > With error policy equals stop, if M has an I/O error, now R also gets
> > prepended to the per device DMA restart queue, which will be retried
> > when VM resumes.  It leads to a double completion (in symptoms of memory
> > corruption or use after free).
> > 
> > Adding R to the queue is superfluous, only M needs to be in the queue.
> > 
> > Fix this by marking request R as "merged" and skipping it in
> > virtio_blk_handle_rw_error.

The commit message is outdated. "merged" requests are actually skipped in
virtio_blk_handle_request.

> > 
> > Cc: qemu-stable@nongnu.org
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > 
> > ---
> > 
> > v2: Don't lose the request in migration. [Paolo]
> > ---
> >  hw/block/virtio-blk.c          | 7 +++++++
> >  include/hw/virtio/virtio-blk.h | 1 +
> >  2 files changed, 8 insertions(+)
> > 
> > diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> > index e70fccf..5cdb06f 100644
> > --- a/hw/block/virtio-blk.c
> > +++ b/hw/block/virtio-blk.c
> > @@ -36,6 +36,7 @@ VirtIOBlockReq *virtio_blk_alloc_request(VirtIOBlock *s)
> >      req->in_len = 0;
> >      req->next = NULL;
> >      req->mr_next = NULL;
> > +    req->merged = false;
> >      return req;
> >  }
> >  
> > @@ -344,6 +345,7 @@ static inline void submit_requests(BlockBackend *blk, MultiReqBuffer *mrb,
> >          for (i = start + 1; i < start + num_reqs; i++) {
> >              qemu_iovec_concat(qiov, &mrb->reqs[i]->qiov, 0,
> >                                mrb->reqs[i]->qiov.size);
> > +            mrb->reqs[i]->merged = true;
> >              mrb->reqs[i - 1]->mr_next = mrb->reqs[i];
> >              nb_sectors += mrb->reqs[i]->qiov.size / BDRV_SECTOR_SIZE;
> >          }
> > @@ -511,6 +513,11 @@ void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
> >                - sizeof(struct virtio_blk_inhdr);
> >      iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
> >  
> > +    if (req->merged) {
> > +        /* Enough for restarting a (migrated) merged request, no need to
> > +         * actually submit I/O. */
> > +        return;
> > +    }

This is not enough.

There is a risk that the coalesced requests being restarted here are also
merged, which will leak the requests originally merged into it.

> >      type = virtio_ldl_p(VIRTIO_DEVICE(req->dev), &req->out.type);
> >  
> >      /* VIRTIO_BLK_T_OUT defines the command direction. VIRTIO_BLK_T_BARRIER
> > diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
> > index 6bf5905..db4adf4 100644
> > --- a/include/hw/virtio/virtio-blk.h
> > +++ b/include/hw/virtio/virtio-blk.h
> > @@ -70,6 +70,7 @@ typedef struct VirtIOBlockReq {
> >      size_t in_len;
> >      struct VirtIOBlockReq *next;
> >      struct VirtIOBlockReq *mr_next;
> > +    bool merged;
> >      BlockAcctCookie acct;
> >  } VirtIOBlockReq;
> 
> I'm not sure if this patch truly fixes the bug:
> 
> virtio_blk_rw_complete() doesn't do req->mr_next = NULL.  There is a
> potential double-free if resubmitting failed requests doesn't overwrite
> themr_next field.
> 
> This can be fixed by adding req->mr_next = NULL to the loop in
> virtio_blk_rw_complete().
> 
> Is that enough to solve the bug?  I don't think adding a new field is
> necessary.
> 
> If not, please explain the double-free.

The first free is the expected one when the coalesced request is completed.
The second free is because virtio_blk_rw_complete was also called on requests
who have "merged == true", which is a mistake in virtio_blk_dma_restart_bh.

I don't think adding req->mr_next in virtio_blk_rw_complete in the I/O error
path is right or helpful.

Fam

Re: [Qemu-devel] [PATCH v2] virtio-blk: Fix double completion for werror=stop

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5219 bytes --]

On Tue, Nov 17, 2015 at 03:28:29PM +0800, Fam Zheng wrote:
> On Tue, 11/17 14:58, Stefan Hajnoczi wrote:
> > On Mon, Nov 16, 2015 at 02:10:36PM +0800, Fam Zheng wrote:
> > > When a request R is absorbed by request M, it is appended to the
> > > "mr_next" queue led by M, and is completed together with the completion
> > > of M, in virtio_blk_rw_complete.
> > > 
> > > With error policy equals stop, if M has an I/O error, now R also gets
> > > prepended to the per device DMA restart queue, which will be retried
> > > when VM resumes.  It leads to a double completion (in symptoms of memory
> > > corruption or use after free).
> > > 
> > > Adding R to the queue is superfluous, only M needs to be in the queue.
> > > 
> > > Fix this by marking request R as "merged" and skipping it in
> > > virtio_blk_handle_rw_error.
> 
> The commit message is outdated. "merged" requests are actually skipped in
> virtio_blk_handle_request.
> 
> > > 
> > > Cc: qemu-stable@nongnu.org
> > > Signed-off-by: Fam Zheng <famz@redhat.com>
> > > 
> > > ---
> > > 
> > > v2: Don't lose the request in migration. [Paolo]
> > > ---
> > >  hw/block/virtio-blk.c          | 7 +++++++
> > >  include/hw/virtio/virtio-blk.h | 1 +
> > >  2 files changed, 8 insertions(+)
> > > 
> > > diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> > > index e70fccf..5cdb06f 100644
> > > --- a/hw/block/virtio-blk.c
> > > +++ b/hw/block/virtio-blk.c
> > > @@ -36,6 +36,7 @@ VirtIOBlockReq *virtio_blk_alloc_request(VirtIOBlock *s)
> > >      req->in_len = 0;
> > >      req->next = NULL;
> > >      req->mr_next = NULL;
> > > +    req->merged = false;
> > >      return req;
> > >  }
> > >  
> > > @@ -344,6 +345,7 @@ static inline void submit_requests(BlockBackend *blk, MultiReqBuffer *mrb,
> > >          for (i = start + 1; i < start + num_reqs; i++) {
> > >              qemu_iovec_concat(qiov, &mrb->reqs[i]->qiov, 0,
> > >                                mrb->reqs[i]->qiov.size);
> > > +            mrb->reqs[i]->merged = true;
> > >              mrb->reqs[i - 1]->mr_next = mrb->reqs[i];
> > >              nb_sectors += mrb->reqs[i]->qiov.size / BDRV_SECTOR_SIZE;
> > >          }
> > > @@ -511,6 +513,11 @@ void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
> > >                - sizeof(struct virtio_blk_inhdr);
> > >      iov_discard_back(in_iov, &in_num, sizeof(struct virtio_blk_inhdr));
> > >  
> > > +    if (req->merged) {
> > > +        /* Enough for restarting a (migrated) merged request, no need to
> > > +         * actually submit I/O. */
> > > +        return;
> > > +    }
> 
> This is not enough.
> 
> There is a risk that the coalesced requests being restarted here are also
> merged, which will leak the requests originally merged into it.
> 
> > >      type = virtio_ldl_p(VIRTIO_DEVICE(req->dev), &req->out.type);
> > >  
> > >      /* VIRTIO_BLK_T_OUT defines the command direction. VIRTIO_BLK_T_BARRIER
> > > diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
> > > index 6bf5905..db4adf4 100644
> > > --- a/include/hw/virtio/virtio-blk.h
> > > +++ b/include/hw/virtio/virtio-blk.h
> > > @@ -70,6 +70,7 @@ typedef struct VirtIOBlockReq {
> > >      size_t in_len;
> > >      struct VirtIOBlockReq *next;
> > >      struct VirtIOBlockReq *mr_next;
> > > +    bool merged;
> > >      BlockAcctCookie acct;
> > >  } VirtIOBlockReq;
> > 
> > I'm not sure if this patch truly fixes the bug:
> > 
> > virtio_blk_rw_complete() doesn't do req->mr_next = NULL.  There is a
> > potential double-free if resubmitting failed requests doesn't overwrite
> > themr_next field.
> > 
> > This can be fixed by adding req->mr_next = NULL to the loop in
> > virtio_blk_rw_complete().
> > 
> > Is that enough to solve the bug?  I don't think adding a new field is
> > necessary.
> > 
> > If not, please explain the double-free.
> 
> The first free is the expected one when the coalesced request is completed.
> The second free is because virtio_blk_rw_complete was also called on requests
> who have "merged == true", which is a mistake in virtio_blk_dma_restart_bh.
> 
> I don't think adding req->mr_next in virtio_blk_rw_complete in the I/O error
> path is right or helpful.

Question about your patch: How is the merged qiov retained across
restart?  The merged qiov is freed the first time
virtio_blk_rw_complete() with an error gets called and you skip
processing merged requests in virtio_blk_handle_request().

I think the result of your patch is that only the head request is
resubmitted to the host OS.  Non-head requests are not submitted to the
host OS.  The requests all complete at the virtio-blk level but only the
head request actually gets data transferred.

Like I said, I think the fix is much simpler.  Forget about adding a
merged field and just dissolve the relationship between merged requests
in virtio_blk_rw_complete() by setting req->mr_next = NULL in
virtio_blk_rw_complete()'s loop.  The code was designed to work this way
because virtio_blk_dma_restart_bh() parses and submits requests again
from scratch at the virtio-blk level.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]