* [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
@ 2013-10-25 15:01 Jack Wang
2013-10-28 9:15 ` Jack Wang
2013-10-30 9:50 ` Stefan Hajnoczi
0 siblings, 2 replies; 6+ messages in thread
From: Jack Wang @ 2013-10-25 15:01 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, Alexey Zaytsev, Stefan Hajnoczi
Hi Experts,
We've seen guest block io lost in a VM.any response will be helpful
environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk
the exported vdb has very high infight io over 300. Some times later a
lot io process in D state, looks a lot requests is lost in below storage
stack.
We're use qemu-kvm 1.0, host kernel 3.4.51
In qemu log of virtio-blk.c
I found below commit, I wonder is it possible the workload generate some
unknown reqests to qemu that lost in virtio_blk_handle_read?
I do some fio test myself, I cann't generate so call unknown request type.
Any response will be helpful.
Jack
commit 9e72c45033770b81b536ac6091e91807247cc25a
Author: Alexey Zaytsev <alexey.zaytsev@gmail.com>
Date: Thu Dec 13 09:03:43 2012 +0200
virtio-blk: Return UNSUPP for unknown request types
Currently, all unknown requests are treated as VIRTIO_BLK_T_IN
Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c
index 92c745a..df57b35 100644
--- a/hw/virtio-blk.c
+++ b/hw/virtio-blk.c
@@ -398,10 +398,14 @@ static void
virtio_blk_handle_request(VirtIOBlockReq *req,
qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
req->elem.out_num - 1);
virtio_blk_handle_write(req, mrb);
- } else {
+ } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
+ /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
req->elem.in_num - 1);
virtio_blk_handle_read(req);
+ } else {
+ virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+ g_free(req);
}
}
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
2013-10-25 15:01 [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? Jack Wang
@ 2013-10-28 9:15 ` Jack Wang
2013-10-28 9:54 ` Alexey Zaytsev
2013-10-30 9:50 ` Stefan Hajnoczi
1 sibling, 1 reply; 6+ messages in thread
From: Jack Wang @ 2013-10-28 9:15 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, Alexey Zaytsev, Stefan Hajnoczi
Hello Kevin & Stefan
Any comments or wild guess about the bug?
Regards,
Jack
On 10/25/2013 05:01 PM, Jack Wang wrote:
> Hi Experts,
>
> We've seen guest block io lost in a VM.any response will be helpful
>
> environment is:
> guest os: Ubuntu 1304
> running busy database workload with xfs on a disk export with virtio-blk
>
> the exported vdb has very high infight io over 300. Some times later a
> lot io process in D state, looks a lot requests is lost in below storage
> stack.
>
> We're use qemu-kvm 1.0, host kernel 3.4.51
>
> In qemu log of virtio-blk.c
> I found below commit, I wonder is it possible the workload generate some
> unknown reqests to qemu that lost in virtio_blk_handle_read?
> I do some fio test myself, I cann't generate so call unknown request type.
>
> Any response will be helpful.
>
> Jack
>
>
> commit 9e72c45033770b81b536ac6091e91807247cc25a
> Author: Alexey Zaytsev <alexey.zaytsev@gmail.com>
> Date: Thu Dec 13 09:03:43 2012 +0200
>
> virtio-blk: Return UNSUPP for unknown request types
>
> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN
>
> Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>
> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c
> index 92c745a..df57b35 100644
> --- a/hw/virtio-blk.c
> +++ b/hw/virtio-blk.c
> @@ -398,10 +398,14 @@ static void
> virtio_blk_handle_request(VirtIOBlockReq *req,
> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
> req->elem.out_num - 1);
> virtio_blk_handle_write(req, mrb);
> - } else {
> + } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
> + /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
> req->elem.in_num - 1);
> virtio_blk_handle_read(req);
> + } else {
> + virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
> + g_free(req);
> }
> }
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
2013-10-28 9:15 ` Jack Wang
@ 2013-10-28 9:54 ` Alexey Zaytsev
2013-10-28 10:13 ` Jack Wang
0 siblings, 1 reply; 6+ messages in thread
From: Alexey Zaytsev @ 2013-10-28 9:54 UTC (permalink / raw)
To: Jack Wang; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi
Hey.
I very much doubt this commit could be causing the problem, as qemu
would never set wrong request type in the first place. You can easily
check by either reverting it, or adding a printk() before
virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP).
On Mon, Oct 28, 2013 at 10:15 AM, Jack Wang <xjtuwjp@gmail.com> wrote:
> Hello Kevin & Stefan
>
> Any comments or wild guess about the bug?
>
> Regards,
> Jack
>
> On 10/25/2013 05:01 PM, Jack Wang wrote:
>> Hi Experts,
>>
>> We've seen guest block io lost in a VM.any response will be helpful
>>
>> environment is:
>> guest os: Ubuntu 1304
>> running busy database workload with xfs on a disk export with virtio-blk
>>
>> the exported vdb has very high infight io over 300. Some times later a
>> lot io process in D state, looks a lot requests is lost in below storage
>> stack.
>>
>> We're use qemu-kvm 1.0, host kernel 3.4.51
>>
>> In qemu log of virtio-blk.c
>> I found below commit, I wonder is it possible the workload generate some
>> unknown reqests to qemu that lost in virtio_blk_handle_read?
>> I do some fio test myself, I cann't generate so call unknown request type.
>>
>> Any response will be helpful.
>>
>> Jack
>>
>>
>> commit 9e72c45033770b81b536ac6091e91807247cc25a
>> Author: Alexey Zaytsev <alexey.zaytsev@gmail.com>
>> Date: Thu Dec 13 09:03:43 2012 +0200
>>
>> virtio-blk: Return UNSUPP for unknown request types
>>
>> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN
>>
>> Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>>
>> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c
>> index 92c745a..df57b35 100644
>> --- a/hw/virtio-blk.c
>> +++ b/hw/virtio-blk.c
>> @@ -398,10 +398,14 @@ static void
>> virtio_blk_handle_request(VirtIOBlockReq *req,
>> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
>> req->elem.out_num - 1);
>> virtio_blk_handle_write(req, mrb);
>> - } else {
>> + } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
>> + /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
>> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
>> req->elem.in_num - 1);
>> virtio_blk_handle_read(req);
>> + } else {
>> + virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
>> + g_free(req);
>> }
>> }
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
2013-10-28 9:54 ` Alexey Zaytsev
@ 2013-10-28 10:13 ` Jack Wang
0 siblings, 0 replies; 6+ messages in thread
From: Jack Wang @ 2013-10-28 10:13 UTC (permalink / raw)
To: Alexey Zaytsev; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi
On 10/28/2013 10:54 AM, Alexey Zaytsev wrote:
> Hey.
>
> I very much doubt this commit could be causing the problem, as qemu
> would never set wrong request type in the first place. You can easily
> check by either reverting it, or adding a printk() before
> virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP).
Hi Alexey,
Thanks for you input.
According to my test results, yes, as you said, virtio-blk never
generate wrong request type. So the commit is only a small cosmetic
extra check:(
As there's nothing abnormal in host server and storage, there must be
some hidden bug somewhere, damn it.
Regards,
Jack
>
> On Mon, Oct 28, 2013 at 10:15 AM, Jack Wang <xjtuwjp@gmail.com> wrote:
>> Hello Kevin & Stefan
>>
>> Any comments or wild guess about the bug?
>>
>> Regards,
>> Jack
>>
>> On 10/25/2013 05:01 PM, Jack Wang wrote:
>>> Hi Experts,
>>>
>>> We've seen guest block io lost in a VM.any response will be helpful
>>>
>>> environment is:
>>> guest os: Ubuntu 1304
>>> running busy database workload with xfs on a disk export with virtio-blk
>>>
>>> the exported vdb has very high infight io over 300. Some times later a
>>> lot io process in D state, looks a lot requests is lost in below storage
>>> stack.
>>>
>>> We're use qemu-kvm 1.0, host kernel 3.4.51
>>>
>>> In qemu log of virtio-blk.c
>>> I found below commit, I wonder is it possible the workload generate some
>>> unknown reqests to qemu that lost in virtio_blk_handle_read?
>>> I do some fio test myself, I cann't generate so call unknown request type.
>>>
>>> Any response will be helpful.
>>>
>>> Jack
>>>
>>>
>>> commit 9e72c45033770b81b536ac6091e91807247cc25a
>>> Author: Alexey Zaytsev <alexey.zaytsev@gmail.com>
>>> Date: Thu Dec 13 09:03:43 2012 +0200
>>>
>>> virtio-blk: Return UNSUPP for unknown request types
>>>
>>> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN
>>>
>>> Signed-off-by: Alexey Zaytsev <alexey.zaytsev@gmail.com>
>>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>>>
>>> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c
>>> index 92c745a..df57b35 100644
>>> --- a/hw/virtio-blk.c
>>> +++ b/hw/virtio-blk.c
>>> @@ -398,10 +398,14 @@ static void
>>> virtio_blk_handle_request(VirtIOBlockReq *req,
>>> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
>>> req->elem.out_num - 1);
>>> virtio_blk_handle_write(req, mrb);
>>> - } else {
>>> + } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
>>> + /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
>>> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
>>> req->elem.in_num - 1);
>>> virtio_blk_handle_read(req);
>>> + } else {
>>> + virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
>>> + g_free(req);
>>> }
>>> }
>>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
2013-10-25 15:01 [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? Jack Wang
2013-10-28 9:15 ` Jack Wang
@ 2013-10-30 9:50 ` Stefan Hajnoczi
2013-10-30 10:55 ` Jack Wang
1 sibling, 1 reply; 6+ messages in thread
From: Stefan Hajnoczi @ 2013-10-30 9:50 UTC (permalink / raw)
To: Jack Wang; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Alexey Zaytsev
On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote:
> We've seen guest block io lost in a VM.any response will be helpful
>
> environment is:
> guest os: Ubuntu 1304
> running busy database workload with xfs on a disk export with virtio-blk
>
> the exported vdb has very high infight io over 300. Some times later a
> lot io process in D state, looks a lot requests is lost in below storage
> stack.
Is the image file on a local file system or are you using a network
storage system (e.g. NFS, Gluster, Ceph, Sheepdog)?
If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O
activity? If that number is very low or zero then there may be a
starvation problem. If that number is reasonable then the workload is
simply bottlenecked on disk I/O.
virtio-blk only has 128 descriptors available so it's not possible to
have 300 requests pending at the virtio-blk layer.
If you suspect QEMU, try building qemu.git/master from source in case
the bug has already been fixed.
If you want to trace I/O requests, you might find this blog post on
writing trace analysis scripts useful:
http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html
Stefan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
2013-10-30 9:50 ` Stefan Hajnoczi
@ 2013-10-30 10:55 ` Jack Wang
0 siblings, 0 replies; 6+ messages in thread
From: Jack Wang @ 2013-10-30 10:55 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Alexey Zaytsev
On 10/30/2013 10:50 AM, Stefan Hajnoczi wrote:
> On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote:
>> We've seen guest block io lost in a VM.any response will be helpful
>>
>> environment is:
>> guest os: Ubuntu 1304
>> running busy database workload with xfs on a disk export with virtio-blk
>>
>> the exported vdb has very high infight io over 300. Some times later a
>> lot io process in D state, looks a lot requests is lost in below storage
>> stack.
>
> Is the image file on a local file system or are you using a network
> storage system (e.g. NFS, Gluster, Ceph, Sheepdog)?
>
> If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O
> activity? If that number is very low or zero then there may be a
> starvation problem. If that number is reasonable then the workload is
> simply bottlenecked on disk I/O.
>
> virtio-blk only has 128 descriptors available so it's not possible to
> have 300 requests pending at the virtio-blk layer.
>
> If you suspect QEMU, try building qemu.git/master from source in case
> the bug has already been fixed.
>
> If you want to trace I/O requests, you might find this blog post on
> writing trace analysis scripts useful:
> http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html
>
> Stefan
>
Thanks Stefan for your valuable input.
The image is on device exported with InfiniBand srp/srpt.
Will follow your suggestions to do further investigation.
The 300 infight ios I memtioned is from the /proc/diskstats Field 9 --
# of I/Os currently in progress.
Jack
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-10-30 10:55 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-25 15:01 [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu? Jack Wang
2013-10-28 9:15 ` Jack Wang
2013-10-28 9:54 ` Alexey Zaytsev
2013-10-28 10:13 ` Jack Wang
2013-10-30 9:50 ` Stefan Hajnoczi
2013-10-30 10:55 ` Jack Wang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).