Re: Interested in contributing to QEMU

[John Snow <jsnow@redhat.com>]

On 2/12/21 7:10 AM, Niteesh G. S. wrote:
> Hello John,
> 
> On Wed, Feb 10, 2021 at 10:33 PM John Snow <jsnow@redhat.com 
> <mailto:jsnow@redhat.com>> wrote:
> 

[...]

> 
>     OK: I think I need to be careful about "issuing" work to someone who
>     isn't (yet) accepted into the program -- I shouldn't misrepresent this
>     arrangement -- but I can give you some more research tips that may help
>     you find your footing.
> 
>     We can work on getting to know QMP a bit better; it sounds like it'd be
>     relevant for both projects.
> 
>     Try using '-qmp qmp.sock,server,nowait' when you boot up QEMU and then
>     open the qmp.sock file with socat and try messing with it.
> 
>     Try going to ./qapi/ in the source tree and "git grep event" to find
>     some event definitions. try grepping for that event name in the QEMU
>     tree at large and try to work out how QEMU emits these events.
> 
>     Try *adding* an event somewhere in ./qapi/ and modifying QEMU to emit
>     this event. Try using rlwrap and socat to connect to this QMP socket
>     and
>     verify that your event is being sent over the wire. Depending on where
>     you add the event, it may be helpful to start QEMU in a paused state
>     and
>     issue a resume command from QMP to make sure you have time to
>     connect to
>     the socket.
> 
> I added an event that is emitted every keypress. It also sends the qcode 
> of the key pressed.
> https://github.com/gs-niteesh/qemu/pull/1/commits/1684f1f499e67ec4af5f75d0cc09eb1ecf9be709 
> <https://github.com/gs-niteesh/qemu/pull/1/commits/1684f1f499e67ec4af5f75d0cc09eb1ecf9be709>
> After doing this, I think I have a basic understanding of how the event 
> stuff works and also
> what happens when a command is issued from the shell.

Good!

> I have summarized my understanding below, please correct me if I am 
> wrong anywhere.

> 1) The JSON files under qemu/qapi define the QMP messages. These JSON 
> files are then
> compiled into their C representations during the build time.

Yep! They are processed using the scripts/qapi/ python package, the QAPI 
generator, which builds those definitions into concrete C types, wires 
up the command handlers for the QMP socket, and so on.

> 2) These are then registered using qmp_register_command.

Yeah, but in most cases the code generator actually handles this part 
for you. Every QMP command we have is (to my knowledge) defined as a 
'command' in the JSON files.

All one has to do as the implementor is to write a function named 
"qmp_my_command_name" with the correct arguments and types. (And make 
sure it gets linked into the final binary!)

> 3) When a QMP request is initiated, the commands respective C function 
> is found from the
> registered list of commands and is dispatched. One of the main functions 
> responsible for the
> finding and dispatching of qmp_* functions is qmp_dispatch in 
> qapi/qmp-dispatch.c. Which
> is invoked from monitor/qmp.c upon a QMP request. The result is then 
> published to the user
> through monitor_qmp_respond -> qmp_send_response.

I believe so. I haven't actually looked too closely at how the API 
wrangler/dispatcher code works. The high-level abstraction is actually 
pretty good in most cases.

> Similar stuff happens to events too, the difference being, the function 
> definitions are autogenerated
> depending on their schema, whereas for commands only the function 
> prototypes are generated.
> The events can be emitted through qapi_event_send_EVENT_NAME functions. 
> These functions
> convert C to JSON and call qapi_event_emit which then finally calls the 
> monitor function to display an
> event to the user.
> 

Yep, exactly.

> Few things I couldn't figure out and understand are
> 1) Who informs the monitor that the command is been issued? Since there 
> are many ways to
> connect to the monitor, I think there should be a generic API which 
> informs about input data.
> But I can't find the code behind all this.

Yes. monitor/qmp.c handles the implementation of the monitor, but 
doesn't specify the actual transports for how the data arrives.

We rely on the chardev for that. See the signature for monitor_init_qmp, 
which receives a "chardev" and then uses it to start listening for data.

> Before sending this mail, I decided to give this another try and found
> qemu_chr_fe_set_handlers is responsible for register the handlers for 
> input and output. Is that
> right? And also does chr_fe mean character_front_end?
> 

QEMU likes to separate out the idea of a frontend and a backend to a 
device. A virtual device presented to a guest as some specific model of 
hardware is the frontend.

The way we achieve the functionality of that device on the user's 
physical machine is the device backend. For character devices, 
(chardev), the backend might just be stdio on your terminal.

...QMP is acting kind of like a frontend device in that it's a user of a 
backend, but it's not really a guest-facing device. FE/BE distinctions 
get kind of weird at times for that reason.

> 2) I couldn't understand QEMU co-routines in general. In this case, I 
> couldn't understand the
> coroutine code in qmp_dispatch in qapi/qmp-dispatch.c. Also please 
> explain OOB in simple
> terms? How are they different from normal commands? And also their 
> relation with coroutines.
> 

Coroutines are a type of "cooperative" user scheduling. In Linux, this 
means that the user process itself switching between tasks at its own 
discretion, independent of the Linux scheduler.

It is "cooperative" because each task needs to indicate when it is ready 
to yield, they are not pre-empted. When a task needs to wait for an 
operation to be ready, it can yield to allow another task to run.

In C on Linux, you could implement this with stackswaps (makecontext, 
swapcontext, setcontext and so on), but there are other ways, and other 
OSes that need an implementation. Look at `coroutine-ucontext.c` for one 
possible implementation.

QMP uses them to dispatch certain commands; those marked as such in 
those qapi JSON files. Look at the definition for 'block_resize'.

OOB commands are complicated, check the docs/devel/qmp-spec.txt file for 
more info on those. Simply (with lots of caveats and gotchas) is that 
QMP commands are processed one at a time in FIFO order. OOB commands, 
however, will "skip the queue" and may be processed immediately. That's 
roughly why "OOB" exists; it's to send critical commands like "cancel" 
or "exit" to try and un-jam QEMU when it gets stuck processing a 
previous command.

Because of this, the handling over what threading context we execute 
those commands in is handled a little differently / more carefully.

> Thanks,
> Niteesh.
> 

--js