* [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
@ 2013-10-25 15:01 Jack Wang
2013-10-28 9:15 ` Jack Wang
2013-10-30 9:50 ` Stefan Hajnoczi
0 siblings, 2 replies; 6+ messages in thread
From: Jack Wang @ 2013-10-25 15:01 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, Alexey Zaytsev, Stefan Hajnoczi
Hi Experts,
We've seen guest block io lost in a VM.any response will be helpful
environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk
the exported vdb has very high infight io over 300. Some times later a
lot io process in D state, looks a lot requests is lost in below storage
stack.
We're use qemu-kvm 1.0, host kernel 3.4.51
In qemu log of virtio-blk.c
I found below commit, I wonder is it possible the workload generate some
unknown reqests to qemu that lost in virtio_blk_handle_read?
I do some fio test myself, I cann't generate so call unknown request type.
Any response will be helpful.
Jack
commit 9e72c45033770b81b536ac6091e91807247cc25a
Author: Alexey Zaytsev <alexey.zaytsev@gmail.com>
Date: Thu Dec 13 09:03:43 2012 +0200
virtio-blk: Return UNSUPP for unknown request types
Currently, all unknown requests are treated as VIRTIO_BLK_T_IN
Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c
index 92c745a..df57b35 100644
--- a/hw/virtio-blk.c
+++ b/hw/virtio-blk.c
@@ -398,10 +398,14 @@ static void
virtio_blk_handle_request(VirtIOBlockReq *req,
qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
req->elem.out_num - 1);
virtio_blk_handle_write(req, mrb);
- } else {
+ } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
+ /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
req->elem.in_num - 1);
virtio_blk_handle_read(req);
+ } else {
+ virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+ g_free(req);
}
}
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? 2013-10-25 15:01 [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? Jack Wang @ 2013-10-28 9:15 ` Jack Wang 2013-10-28 9:54 ` Alexey Zaytsev 2013-10-30 9:50 ` Stefan Hajnoczi 1 sibling, 1 reply; 6+ messages in thread From: Jack Wang @ 2013-10-28 9:15 UTC (permalink / raw) To: qemu-devel; +Cc: Kevin Wolf, Alexey Zaytsev, Stefan Hajnoczi Hello Kevin & Stefan Any comments or wild guess about the bug? Regards, Jack On 10/25/2013 05:01 PM, Jack Wang wrote: > Hi Experts, > > We've seen guest block io lost in a VM.any response will be helpful > > environment is: > guest os: Ubuntu 1304 > running busy database workload with xfs on a disk export with virtio-blk > > the exported vdb has very high infight io over 300. Some times later a > lot io process in D state, looks a lot requests is lost in below storage > stack. > > We're use qemu-kvm 1.0, host kernel 3.4.51 > > In qemu log of virtio-blk.c > I found below commit, I wonder is it possible the workload generate some > unknown reqests to qemu that lost in virtio_blk_handle_read? > I do some fio test myself, I cann't generate so call unknown request type. > > Any response will be helpful. > > Jack > > > commit 9e72c45033770b81b536ac6091e91807247cc25a > Author: Alexey Zaytsev <alexey.zaytsev@gmail.com> > Date: Thu Dec 13 09:03:43 2012 +0200 > > virtio-blk: Return UNSUPP for unknown request types > > Currently, all unknown requests are treated as VIRTIO_BLK_T_IN > > Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > > diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c > index 92c745a..df57b35 100644 > --- a/hw/virtio-blk.c > +++ b/hw/virtio-blk.c > @@ -398,10 +398,14 @@ static void > virtio_blk_handle_request(VirtIOBlockReq *req, > qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], > req->elem.out_num - 1); > virtio_blk_handle_write(req, mrb); > - } else { > + } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { > + /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ > qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], > req->elem.in_num - 1); > virtio_blk_handle_read(req); > + } else { > + virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); > + g_free(req); > } > } > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? 2013-10-28 9:15 ` Jack Wang @ 2013-10-28 9:54 ` Alexey Zaytsev 2013-10-28 10:13 ` Jack Wang 0 siblings, 1 reply; 6+ messages in thread From: Alexey Zaytsev @ 2013-10-28 9:54 UTC (permalink / raw) To: Jack Wang; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi Hey. I very much doubt this commit could be causing the problem, as qemu would never set wrong request type in the first place. You can easily check by either reverting it, or adding a printk() before virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP). On Mon, Oct 28, 2013 at 10:15 AM, Jack Wang <xjtuwjp@gmail.com> wrote: > Hello Kevin & Stefan > > Any comments or wild guess about the bug? > > Regards, > Jack > > On 10/25/2013 05:01 PM, Jack Wang wrote: >> Hi Experts, >> >> We've seen guest block io lost in a VM.any response will be helpful >> >> environment is: >> guest os: Ubuntu 1304 >> running busy database workload with xfs on a disk export with virtio-blk >> >> the exported vdb has very high infight io over 300. Some times later a >> lot io process in D state, looks a lot requests is lost in below storage >> stack. >> >> We're use qemu-kvm 1.0, host kernel 3.4.51 >> >> In qemu log of virtio-blk.c >> I found below commit, I wonder is it possible the workload generate some >> unknown reqests to qemu that lost in virtio_blk_handle_read? >> I do some fio test myself, I cann't generate so call unknown request type. >> >> Any response will be helpful. >> >> Jack >> >> >> commit 9e72c45033770b81b536ac6091e91807247cc25a >> Author: Alexey Zaytsev <alexey.zaytsev@gmail.com> >> Date: Thu Dec 13 09:03:43 2012 +0200 >> >> virtio-blk: Return UNSUPP for unknown request types >> >> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN >> >> Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> >> >> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c >> index 92c745a..df57b35 100644 >> --- a/hw/virtio-blk.c >> +++ b/hw/virtio-blk.c >> @@ -398,10 +398,14 @@ static void >> virtio_blk_handle_request(VirtIOBlockReq *req, >> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], >> req->elem.out_num - 1); >> virtio_blk_handle_write(req, mrb); >> - } else { >> + } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { >> + /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ >> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], >> req->elem.in_num - 1); >> virtio_blk_handle_read(req); >> + } else { >> + virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); >> + g_free(req); >> } >> } >> > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? 2013-10-28 9:54 ` Alexey Zaytsev @ 2013-10-28 10:13 ` Jack Wang 0 siblings, 0 replies; 6+ messages in thread From: Jack Wang @ 2013-10-28 10:13 UTC (permalink / raw) To: Alexey Zaytsev; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi On 10/28/2013 10:54 AM, Alexey Zaytsev wrote: > Hey. > > I very much doubt this commit could be causing the problem, as qemu > would never set wrong request type in the first place. You can easily > check by either reverting it, or adding a printk() before > virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP). Hi Alexey, Thanks for you input. According to my test results, yes, as you said, virtio-blk never generate wrong request type. So the commit is only a small cosmetic extra check:( As there's nothing abnormal in host server and storage, there must be some hidden bug somewhere, damn it. Regards, Jack > > On Mon, Oct 28, 2013 at 10:15 AM, Jack Wang <xjtuwjp@gmail.com> wrote: >> Hello Kevin & Stefan >> >> Any comments or wild guess about the bug? >> >> Regards, >> Jack >> >> On 10/25/2013 05:01 PM, Jack Wang wrote: >>> Hi Experts, >>> >>> We've seen guest block io lost in a VM.any response will be helpful >>> >>> environment is: >>> guest os: Ubuntu 1304 >>> running busy database workload with xfs on a disk export with virtio-blk >>> >>> the exported vdb has very high infight io over 300. Some times later a >>> lot io process in D state, looks a lot requests is lost in below storage >>> stack. >>> >>> We're use qemu-kvm 1.0, host kernel 3.4.51 >>> >>> In qemu log of virtio-blk.c >>> I found below commit, I wonder is it possible the workload generate some >>> unknown reqests to qemu that lost in virtio_blk_handle_read? >>> I do some fio test myself, I cann't generate so call unknown request type. >>> >>> Any response will be helpful. >>> >>> Jack >>> >>> >>> commit 9e72c45033770b81b536ac6091e91807247cc25a >>> Author: Alexey Zaytsev <alexey.zaytsev@gmail.com> >>> Date: Thu Dec 13 09:03:43 2012 +0200 >>> >>> virtio-blk: Return UNSUPP for unknown request types >>> >>> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN >>> >>> Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com> >>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> >>> >>> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c >>> index 92c745a..df57b35 100644 >>> --- a/hw/virtio-blk.c >>> +++ b/hw/virtio-blk.c >>> @@ -398,10 +398,14 @@ static void >>> virtio_blk_handle_request(VirtIOBlockReq *req, >>> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], >>> req->elem.out_num - 1); >>> virtio_blk_handle_write(req, mrb); >>> - } else { >>> + } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { >>> + /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ >>> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], >>> req->elem.in_num - 1); >>> virtio_blk_handle_read(req); >>> + } else { >>> + virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); >>> + g_free(req); >>> } >>> } >>> >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? 2013-10-25 15:01 [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? Jack Wang 2013-10-28 9:15 ` Jack Wang @ 2013-10-30 9:50 ` Stefan Hajnoczi 2013-10-30 10:55 ` Jack Wang 1 sibling, 1 reply; 6+ messages in thread From: Stefan Hajnoczi @ 2013-10-30 9:50 UTC (permalink / raw) To: Jack Wang; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Alexey Zaytsev On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote: > We've seen guest block io lost in a VM.any response will be helpful > > environment is: > guest os: Ubuntu 1304 > running busy database workload with xfs on a disk export with virtio-blk > > the exported vdb has very high infight io over 300. Some times later a > lot io process in D state, looks a lot requests is lost in below storage > stack. Is the image file on a local file system or are you using a network storage system (e.g. NFS, Gluster, Ceph, Sheepdog)? If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O activity? If that number is very low or zero then there may be a starvation problem. If that number is reasonable then the workload is simply bottlenecked on disk I/O. virtio-blk only has 128 descriptors available so it's not possible to have 300 requests pending at the virtio-blk layer. If you suspect QEMU, try building qemu.git/master from source in case the bug has already been fixed. If you want to trace I/O requests, you might find this blog post on writing trace analysis scripts useful: http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html Stefan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? 2013-10-30 9:50 ` Stefan Hajnoczi @ 2013-10-30 10:55 ` Jack Wang 0 siblings, 0 replies; 6+ messages in thread From: Jack Wang @ 2013-10-30 10:55 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Alexey Zaytsev On 10/30/2013 10:50 AM, Stefan Hajnoczi wrote: > On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote: >> We've seen guest block io lost in a VM.any response will be helpful >> >> environment is: >> guest os: Ubuntu 1304 >> running busy database workload with xfs on a disk export with virtio-blk >> >> the exported vdb has very high infight io over 300. Some times later a >> lot io process in D state, looks a lot requests is lost in below storage >> stack. > > Is the image file on a local file system or are you using a network > storage system (e.g. NFS, Gluster, Ceph, Sheepdog)? > > If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O > activity? If that number is very low or zero then there may be a > starvation problem. If that number is reasonable then the workload is > simply bottlenecked on disk I/O. > > virtio-blk only has 128 descriptors available so it's not possible to > have 300 requests pending at the virtio-blk layer. > > If you suspect QEMU, try building qemu.git/master from source in case > the bug has already been fixed. > > If you want to trace I/O requests, you might find this blog post on > writing trace analysis scripts useful: > http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html > > Stefan > Thanks Stefan for your valuable input. The image is on device exported with InfiniBand srp/srpt. Will follow your suggestions to do further investigation. The 300 infight ios I memtioned is from the /proc/diskstats Field 9 -- # of I/Os currently in progress. Jack ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-10-30 10:55 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-10-25 15:01 [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? Jack Wang 2013-10-28 9:15 ` Jack Wang 2013-10-28 9:54 ` Alexey Zaytsev 2013-10-28 10:13 ` Jack Wang 2013-10-30 9:50 ` Stefan Hajnoczi 2013-10-30 10:55 ` Jack Wang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).