[Qemu-devel] qemu virtIO blocking operation - question

[Qemu-devel] qemu virtIO blocking operation - question

[Sinha, Ani]

Hey guys,

So I am writing a virtIO driver for a device that supports blocking calls like poll() etc. Now the front end paravirtualized driver mirrors the request to the backend "host" qemu process that then does the actual call on the host kernel on behalf of the guest. Now my question is, when I do a blocking call from within the callback that I have registered when I created the virtqueue, does it block the entire vcpu? If it does, do I have to create an async context for it? Looking at virtio-net and virtio-block, I can see the concept of bottom halves but I am not sure if this helps me in any way.

Sorry guys, I am just starting to hack on QEMU, so some of my questions might be newbie-ish.

Any help, pointers will be much appreciated.

Cheers,
Ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Stefan Hajnoczi]

On Fri, Sep 09, 2011 at 07:45:17PM -0500, Sinha, Ani wrote:
> So I am writing a virtIO driver for a device that supports blocking calls like poll() etc. Now the front end paravirtualized driver mirrors the request to the backend "host" qemu process that then does the actual call on the host kernel on behalf of the guest. Now my question is, when I do a blocking call from within the callback that I have registered when I created the virtqueue, does it block the entire vcpu? If it does, do I have to create an async context for it? Looking at virtio-net and virtio-block, I can see the concept of bottom halves but I am not sure if this helps me in any way.

What device are you adding?  It would help to share what you are trying
to accomplish.

QEMU has an event loop where you can register non-blocking file
descriptors.  Callback functions that are invoked when the file
descriptor becomes readable/writable.  This is how block I/O,
networking, VNC UI, monitor, etc work and probably what you should do
too.

You may find these posts I wrote helpful, they explain vcpu threads and
the I/O thread:
http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html
http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html

Stefan

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Sinha, Ani]

On Sep 11, 2011, at 6:34 AM, Stefan Hajnoczi wrote:

> On Fri, Sep 09, 2011 at 07:45:17PM -0500, Sinha, Ani wrote:
>> So I am writing a virtIO driver for a device that supports blocking calls like poll() etc. Now the front end paravirtualized driver mirrors the request to the backend "host" qemu process that then does the actual call on the host kernel on behalf of the guest. Now my question is, when I do a blocking call from within the callback that I have registered when I created the virtqueue, does it block the entire vcpu? If it does, do I have to create an async context for it? Looking at virtio-net and virtio-block, I can see the concept of bottom halves but I am not sure if this helps me in any way.
>
> What device are you adding?  It would help to share what you are trying
> to accomplish.


We are trying to paravirtualize the IPMI device (/dev/ipmi0).


>
> QEMU has an event loop where you can register non-blocking file
> descriptors.  Callback functions that are invoked when the file
> descriptor becomes readable/writable.  This is how block I/O,
> networking, VNC UI, monitor, etc work and probably what you should do
> too.


Hmm, I looked closely at virtio-block driver. It looks like it built on top of posix aio infrastructure (posix-aio-compat.c). I'll probably have to build something similar or use the event loop you suggested.


>
> You may find these posts I wrote helpful, they explain vcpu threads and
> the I/O thread:
> http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html
> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html


Thanks a lot. I will look into these articles.

Cheers,
ani



============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Stefan Hajnoczi]

On Mon, Sep 12, 2011 at 7:31 PM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>
> On Sep 11, 2011, at 6:34 AM, Stefan Hajnoczi wrote:
>
>> On Fri, Sep 09, 2011 at 07:45:17PM -0500, Sinha, Ani wrote:
>>> So I am writing a virtIO driver for a device that supports blocking calls like poll() etc. Now the front end paravirtualized driver mirrors the request to the backend "host" qemu process that then does the actual call on the host kernel on behalf of the guest. Now my question is, when I do a blocking call from within the callback that I have registered when I created the virtqueue, does it block the entire vcpu? If it does, do I have to create an async context for it? Looking at virtio-net and virtio-block, I can see the concept of bottom halves but I am not sure if this helps me in any way.
>>
>> What device are you adding?  It would help to share what you are trying
>> to accomplish.
>
>
> We are trying to paravirtualize the IPMI device (/dev/ipmi0).

>From http://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface:
"An implementation of IPMI version 1.5 can communicate via a direct
serial connection or via a side-band local area network (LAN)
connection to a remote client."

Why do you need a new virtio device?  Can't you use virtio-serial?
This is what other management channels are using for host<->guest
agents.

What features and use cases does paravirtualized IPMI provide?

Stefan

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Sinha, Ani]

>>
>> We are trying to paravirtualize the IPMI device (/dev/ipmi0).
>
> From http://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface:
> "An implementation of IPMI version 1.5 can communicate via a direct
> serial connection or via a side-band local area network (LAN)
> connection to a remote client."
>
> Why do you need a new virtio device?  Can't you use virtio-serial?
> This is what other management channels are using for host<->guest
> agents.


It might be possible. However, we are doing it this way because :

(a) I am not sure of the interactions with the real ipmi device on the host when the device is shared with multiple guests (we are not using pci passthrough to attach the device to a single guest).  The device itself is stateless - that is, it can not handle multiple requests at one time. When multiple users use the device, the driver has to serialize the requests and send it to the device. The existing ipmi driver within Linux (drivers/char/ipmi)  does that now.  But this driver on one guest would not know the existence of another guest. May be it might have to be reworked to work with virtio-serial. Dunno.

So, we wanted to keep this driver as is in the host and build a lightweight interface layer in the guest that can talk to the real driver on the host. Multiple guests would then be like multiple processes accessing the device.

(b) We wanted to gain some experience paravirtualizing devices this way through virtio since we have other proprietary hardware that needs to be paravirtualized.


Makes sense?

Ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Stefan Hajnoczi]

On Tue, Sep 13, 2011 at 6:35 PM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
> Makes sense?

I'm still not sure what the point of multiplexing host IPMI into
guests is.  The guest should not be able to power off the host.  It
shouldn't be able to mess with the watchdog timer.  It may wish to
read sensors but do they even make sense to a guest with virtual
hardware?

Stefan

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Sinha, Ani]

On Sep 15, 2011, at 4:42 AM, Stefan Hajnoczi wrote:

> On Tue, Sep 13, 2011 at 6:35 PM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>> Makes sense?
>
> I'm still not sure what the point of multiplexing host IPMI into
> guests is.  The guest should not be able to power off the host.  It
> shouldn't be able to mess with the watchdog timer.  It may wish to
> read sensors but do they even make sense to a guest with virtual
> hardware?


Those are valid concerns :) However, I do not know how the other teams of this project will make use of this paravirtualized device. My team is supposed to only make this device available in the guest with certain features functional (like sending raw requests and receiving responses). May be later if I get more visibility, I will let you know.

Ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Sinha, Ani]

On Sep 11, 2011, at 6:34 AM, Stefan Hajnoczi wrote:

>
> You may find these posts I wrote helpful, they explain vcpu threads and
> the I/O thread:
> http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html
> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
>


"One example user of worker threads is posix-aio-compat.c, an asynchronous file I/O implementation. When core QEMU issues an aio request it is placed on a queue. Worker threads take requests off the queue and execute them outside of core QEMU. They may perform blocking operations since they execute in their own threads and do not block

Another example is ui/vnc-jobs-async.c which performs compute-intensive image compression and encoding in worker threads."


I wonder why there isn't a general framework for this? Some thread that would take requests off a queue and process them without knowing the internals of the request.

Ani


============================================================
The information contained in this message may be privileged
and confidential and protected from disclosure. If the reader
of this message is not the intended recipient, or an employee
or agent responsible for delivering this message to the
intended recipient, you are hereby notified that any reproduction,
dissemination or distribution of this communication is strictly
prohibited. If you have received this communication in error,
please notify us immediately by replying to the message and
deleting it from your computer. Thank you. Tellabs
============================================================

Re: [Qemu-devel] qemu virtIO blocking operation - question

[Stefan Hajnoczi]

On Mon, Sep 12, 2011 at 10:05 PM, Sinha, Ani <Ani.Sinha@tellabs.com> wrote:
>
> On Sep 11, 2011, at 6:34 AM, Stefan Hajnoczi wrote:
>
>>
>> You may find these posts I wrote helpful, they explain vcpu threads and
>> the I/O thread:
>> http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html
>> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
>>
>
>
> "One example user of worker threads is posix-aio-compat.c, an asynchronous file I/O implementation. When core QEMU issues an aio request it is placed on a queue. Worker threads take requests off the queue and execute them outside of core QEMU. They may perform blocking operations since they execute in their own threads and do not block
>
> Another example is ui/vnc-jobs-async.c which performs compute-intensive image compression and encoding in worker threads."
>
>
> I wonder why there isn't a general framework for this? Some thread that would take requests off a queue and process them without knowing the internals of the request.

There is, you can use GThreadPool.

Stefan