[Qemu-devel] virtqueue corruption in emulation mode?

[Qemu-devel] virtqueue corruption in emulation mode?

[Sinha, Ani]

Hi Guys :

I am using the virtqueue (virtqueue_pop, virtqueue_push etc) in the emulated mode (non-kvm mode) from an IO thread (a separate thread different from main QEMU thread). What I am observing is that the virtqueue memory seems to get corrupt. Either qemu crashes while performing virtqueue_push() (virtqueue_push() -> virtqueue_fill() ->bring_used_idx()->lduw_phys()->qemu_get_ram_ptr()->"bad ram offset") or crashes when the guest accesses a bad memory while using virtqueue. Now this never ever happens when I run QEMU in KVM mode (/dev/kvm present) OR when I use my functions from within the main qemu thread. I am unable to figure out why this is happening. I have looked into my code over and over again and I can't seem to explain this behavior. Can any of you guys give me any inkling?

Thanks a lot in advance.
ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] virtqueue corruption in emulation mode?

[Stefan Hajnoczi]

On Mon, Sep 26, 2011 at 07:16:56PM -0500, Sinha, Ani wrote:
> I am using the virtqueue (virtqueue_pop, virtqueue_push etc) in the emulated mode (non-kvm mode) from an IO thread (a separate thread different from main QEMU thread). What I am observing is that the virtqueue memory seems to get corrupt. Either qemu crashes while performing virtqueue_push() (virtqueue_push() -> virtqueue_fill() ->bring_used_idx()->lduw_phys()->qemu_get_ram_ptr()->"bad ram offset") or crashes when the guest accesses a bad memory while using virtqueue. Now this never ever happens when I run QEMU in KVM mode (/dev/kvm present) OR when I use my functions from within the main qemu thread. I am unable to figure out why this is happening. I have looked into my code over and over again and I can't seem to explain this behavior. Can any of you guys give me any inkling?

QEMU is not thread-safe in general.  It uses a big lock to protect most
of its internal state.

When you say "an IO thread" it sounds like you spawn a new thread
outside the big lock (qemu_global_mutex).  You cannot call the existing
virtqueue functions outside the big lock because they traverse (and
modify!) the memory management data structures.

Please call new threads "helper threads" or something other than "IO
thread" because I/O thread has a specific meaning in QEMU.  It's the
event loop thread that execute main_loop_wait() and dispatches fd
handlers when select(2) returns.  This will prevent confusion.

If you follow the way that existing virtio devices are implemented there
should be no problem.

Stefan

Re: [Qemu-devel] virtqueue corruption in emulation mode?

[Sinha, Ani]

On Sep 27, 2011, at 12:17 AM, Stefan Hajnoczi wrote:

> On Mon, Sep 26, 2011 at 07:16:56PM -0500, Sinha, Ani wrote:
>> I am using the virtqueue (virtqueue_pop, virtqueue_push etc) in the emulated mode (non-kvm mode) from an IO thread (a separate thread different from main QEMU thread). What I am observing is that the virtqueue memory seems to get corrupt. Either qemu crashes while performing virtqueue_push() (virtqueue_push() -> virtqueue_fill() ->bring_used_idx()->lduw_phys()->qemu_get_ram_ptr()->"bad ram offset") or crashes when the guest accesses a bad memory while using virtqueue. Now this never ever happens when I run QEMU in KVM mode (/dev/kvm present) OR when I use my functions from within the main qemu thread. I am unable to figure out why this is happening. I have looked into my code over and over again and I can't seem to explain this behavior. Can any of you guys give me any inkling?
>
> QEMU is not thread-safe in general.  It uses a big lock to protect most
> of its internal state.


I see. So may be I should do something like qemu_set_fd_handler(fd, …) where fd is a pipe and the handler does the virtqueue_push() etc?
Now my question is, is it safe to do elem = virtqueue_pop(vq) from main event loop, then so some work on the elem popped out in an worker thread and then at some later point do a virtqueue_push(vq, elem) from that handler (which is called by main_loop() ->main_loop_wait())?  In other words, the vq reference is being used from the main event loop at two different points from two different functions but not in a contiguous fashion within the same function.


>
> When you say "an IO thread" it sounds like you spawn a new thread
> outside the big lock (qemu_global_mutex).  You cannot call the existing
> virtqueue functions outside the big lock because they traverse (and
> modify!) the memory management data structures.


Thanks for this info.


>
> Please call new threads "helper threads" or something other than "IO
> thread" because I/O thread has a specific meaning in QEMU.  It's the
> event loop thread that execute main_loop_wait() and dispatches fd
> handlers when select(2) returns.  This will prevent confusion.

understood.

>
> If you follow the way that existing virtio devices are implemented there
> should be no problem.
>


yeah, perhaps. It will require a little more thinking.


Ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] virtqueue corruption in emulation mode?

[Stefan Hajnoczi]

On Wed, Sep 28, 2011 at 3:01 AM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>
> On Sep 27, 2011, at 12:17 AM, Stefan Hajnoczi wrote:
>
>> On Mon, Sep 26, 2011 at 07:16:56PM -0500, Sinha, Ani wrote:
>>> I am using the virtqueue (virtqueue_pop, virtqueue_push etc) in the emulated mode (non-kvm mode) from an IO thread (a separate thread different from main QEMU thread). What I am observing is that the virtqueue memory seems to get corrupt. Either qemu crashes while performing virtqueue_push() (virtqueue_push() -> virtqueue_fill() ->bring_used_idx()->lduw_phys()->qemu_get_ram_ptr()->"bad ram offset") or crashes when the guest accesses a bad memory while using virtqueue. Now this never ever happens when I run QEMU in KVM mode (/dev/kvm present) OR when I use my functions from within the main qemu thread. I am unable to figure out why this is happening. I have looked into my code over and over again and I can't seem to explain this behavior. Can any of you guys give me any inkling?
>>
>> QEMU is not thread-safe in general.  It uses a big lock to protect most
>> of its internal state.
>
>
> I see. So may be I should do something like qemu_set_fd_handler(fd, …) where fd is a pipe and the handler does the virtqueue_push() etc?
> Now my question is, is it safe to do elem = virtqueue_pop(vq) from main event loop, then so some work on the elem popped out in an worker thread and then at some later point do a virtqueue_push(vq, elem) from that handler (which is called by main_loop() ->main_loop_wait())?  In other words, the vq reference is being used from the main event loop at two different points from two different functions but not in a contiguous fashion within the same function.

Yes but do you need a helper thread?  Most of QEMU is based on
qemu_set_fd_handler() and callbacks, including for host network and
disk I/O.  If you follow the way QEMU does things it should be
easiest.

Stefan

Re: [Qemu-devel] virtqueue corruption in emulation mode?

[Sinha, Ani]

On Sep 28, 2011, at 1:51 AM, Stefan Hajnoczi wrote:

> On Wed, Sep 28, 2011 at 3:01 AM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>>
>> On Sep 27, 2011, at 12:17 AM, Stefan Hajnoczi wrote:
>>
>>> On Mon, Sep 26, 2011 at 07:16:56PM -0500, Sinha, Ani wrote:
>>>> I am using the virtqueue (virtqueue_pop, virtqueue_push etc) in the emulated mode (non-kvm mode) from an IO thread (a separate thread different from main QEMU thread). What I am observing is that the virtqueue memory seems to get corrupt. Either qemu crashes while performing virtqueue_push() (virtqueue_push() -> virtqueue_fill() ->bring_used_idx()->lduw_phys()->qemu_get_ram_ptr()->"bad ram offset") or crashes when the guest accesses a bad memory while using virtqueue. Now this never ever happens when I run QEMU in KVM mode (/dev/kvm present) OR when I use my functions from within the main qemu thread. I am unable to figure out why this is happening. I have looked into my code over and over again and I can't seem to explain this behavior. Can any of you guys give me any inkling?
>>>
>>> QEMU is not thread-safe in general.  It uses a big lock to protect most
>>> of its internal state.
>>
>>
>> I see. So may be I should do something like qemu_set_fd_handler(fd, …) where fd is a pipe and the handler does the virtqueue_push() etc?
>> Now my question is, is it safe to do elem = virtqueue_pop(vq) from main event loop, then so some work on the elem popped out in an worker thread and then at some later point do a virtqueue_push(vq, elem) from that handler (which is called by main_loop() ->main_loop_wait())?  In other words, the vq reference is being used from the main event loop at two different points from two different functions but not in a contiguous fashion within the same function.
>
> Yes but do you need a helper thread?  Most of QEMU is based on
> qemu_set_fd_handler() and callbacks, including for host network and
> disk I/O.  If you follow the way QEMU does things it should be
> easiest.

I need a helper thread to do blocking IO. The device IOCTLS are inherently blocking, unfortunately.

Ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] virtqueue corruption in emulation mode?

[Stefan Hajnoczi]

On Wed, Sep 28, 2011 at 3:23 PM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>
> On Sep 28, 2011, at 1:51 AM, Stefan Hajnoczi wrote:
>
>> On Wed, Sep 28, 2011 at 3:01 AM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>>>
>>> On Sep 27, 2011, at 12:17 AM, Stefan Hajnoczi wrote:
>>>
>>>> On Mon, Sep 26, 2011 at 07:16:56PM -0500, Sinha, Ani wrote:
>>>>> I am using the virtqueue (virtqueue_pop, virtqueue_push etc) in the emulated mode (non-kvm mode) from an IO thread (a separate thread different from main QEMU thread). What I am observing is that the virtqueue memory seems to get corrupt. Either qemu crashes while performing virtqueue_push() (virtqueue_push() -> virtqueue_fill() ->bring_used_idx()->lduw_phys()->qemu_get_ram_ptr()->"bad ram offset") or crashes when the guest accesses a bad memory while using virtqueue. Now this never ever happens when I run QEMU in KVM mode (/dev/kvm present) OR when I use my functions from within the main qemu thread. I am unable to figure out why this is happening. I have looked into my code over and over again and I can't seem to explain this behavior. Can any of you guys give me any inkling?
>>>>
>>>> QEMU is not thread-safe in general.  It uses a big lock to protect most
>>>> of its internal state.
>>>
>>>
>>> I see. So may be I should do something like qemu_set_fd_handler(fd, …) where fd is a pipe and the handler does the virtqueue_push() etc?
>>> Now my question is, is it safe to do elem = virtqueue_pop(vq) from main event loop, then so some work on the elem popped out in an worker thread and then at some later point do a virtqueue_push(vq, elem) from that handler (which is called by main_loop() ->main_loop_wait())?  In other words, the vq reference is being used from the main event loop at two different points from two different functions but not in a contiguous fashion within the same function.
>>
>> Yes but do you need a helper thread?  Most of QEMU is based on
>> qemu_set_fd_handler() and callbacks, including for host network and
>> disk I/O.  If you follow the way QEMU does things it should be
>> easiest.
>
> I need a helper thread to do blocking IO. The device IOCTLS are inherently blocking, unfortunately.

posix-aio-compat.c already implements a threadpool.  It is geared
towards using the QEMU block layer (BlockDriverState) but the pure
ioctl codepath can be used without a BlockDriverState by passing NULL.
 So this could save you some effort, check out paio_submit().

Stefan

Re: [Qemu-devel] virtqueue corruption in emulation mode?

[Sinha, Ani]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]


On Sep 28, 2011, at 9:47 AM, Stefan Hajnoczi wrote:

I need a helper thread to do blocking IO. The device IOCTLS are inherently blocking, unfortunately.

posix-aio-compat.c already implements a threadpool.  It is geared
towards using the QEMU block layer (BlockDriverState) but the pure
ioctl codepath can be used without a BlockDriverState by passing NULL.
So this could save you some effort, check out paio_submit().


;) yeah, I realized that after looking at the code that I could have used that infrastructure. May be in the next release of this driver I will do that or more realistically when I write another driver, I will make use of it. However, I have one reservation about that code. Like you said, it is geared towards the QEMU block driver. I believe the code can be reworked to be more generic so that it can be reused elsewhere as is.

ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

[-- Attachment #2: Type: text/html, Size: 2217 bytes --]