[Qemu-devel] experience with SDL2, event loop & main thread

[Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

given the reported problems with the SDL integration, here are some things I learned while updating GNU ARM Eclipse QEMU to use SDL2 (SDL1 is already issuing various warnings when running on macOS 10.12, and anyway it has several limitations).

---

initially I expected that the project requires just a simple build script update, but it was much more, I had to completely restructure the graphical code.

in short, the main problem is the competition/contradiction in using the main thread in QEMU.

by tradition, graphical systems and multi-theaded applications do not make good friends, because graphical systems expect to run only on the application's main thread.

SDL2 is not an exception; at least on macOS (Cocoa) and GNU/Linux (openGL), all my attempts to run the graphical event loop on another thread failed with horrible exceptions.


but running the event loop on the main thread is not enough, all graphical primitives must be called from the main thread. 

in QEMU this creates a problem, since normally graphic updates are triggered by the emulated peripherals, which run on the specific emulator thread (for example, in GNU ARM Eclipse QEMU case, for emulated LEDs, GPIO pin changes alternate a rectangle on the board picture).

there might be specific primitives/configurations/platforms where this rule can be relaxed, but the only portable, trouble free, solution is to run everything on the main thread (SDL window creation, draw primitives, event loop, and window destruction).


unfortunately this collides with QEMU use of the main thread to pump the I/O event loop. 

there are two possible solutions:

- move the qemu initialisations and I/O event loop on a new thread, and free the main thread exclusively for the SDL event loop (which can now block waiting for events)
- keep the qemu initialisations and the I/O event loop on the main thread and schedule a timer every few milliseconds to poll if new graphical events are in the queue.


the first solution is obviously the most efficient, since the graphical thread does not consume resources when the there is no graphical activity, and reaction speed to graphical updates is as good as possible. (unfortunately this solution currently does not work on windows)


the second solution, which uses a timer to schedule a poll routine every few milliseconds (for example every 10 ms), is much more inefficient, and adds a small delay to all graphical displays. the timer routine is scheduled to run on the main thread, so the condition to run the graphical event loop on the main thread is satisfied.


fyi, this inefficient second solution, with a timer, is currently used by the qemu graphical consoles.


to guarantee that all graphical primitives will be invoked only from the main thread, calls should not be performed from the actual location (usually the emulator tcg thread), but must be queued as user events and deferred to the main thread, possibly using user events in the graphical event loop; for example:

    case SDL_USEREVENT:

        // User events, enqueued with SDL_PushEvent().
        switch (event->user.code) {

        case GRAPHIC_EVENT_BOARD_INIT:
            board_graphic_context = (BoardGraphicContext *) event->user.data1;

            if (!cortexm_graphic_board_is_graphic_context_initialised(
                    board_graphic_context)) {
                cortexm_graphic_board_init_graphic_context(
                        board_graphic_context);
            }
            break;

        case GRAPHIC_EVENT_LED_INIT:
            state = (GPIOLEDState *) event->user.data1;

            if (!cortexm_graphic_led_is_graphic_context_initialised(
                    &(state->led_graphic_context))) {
                cortexm_graphic_led_init_graphic_context(
                        state->board_graphic_context,
                        &(state->led_graphic_context), state->colour.red,
                        state->colour.green, state->colour.blue);
            }
            break;

        case GRAPHIC_EVENT_LED_TURN:
            state = (GPIOLEDState *) event->user.data1;
            is_on = (bool) event->user.data2;

            cortexm_graphic_led_turn(state->board_graphic_context,
                    &(state->led_graphic_context), is_on);
            break;

	// ...

        default:
            fprintf(stderr, "Unimplemented user event %d\n", event->user.code);
        }
        break;



special care must be observed for window destruction at program termination.

the SDL manuals recommend adding an atexit(SDL_Quit), but this isn't as simple as it looks, because the SDL_Quit must be executed also on the main thread, while the registered atexit() functions are executed on the context of the thread that calls exit(). (trying to run SDL_Quit on another thread hangs on Windows).

in practical terms, this means that exit() can be called only from the main thread. calls from different threads should be replaced with code to push a user event on the graphical loop, and perform the quit on the event loop.


one problematic such case is the semihosting exit. the semihosting code is executed on the emulator thread, so it cannot directly call exit. enqueuing the event is simple, but this code is executed with the io mutex locked, so the I/O loop is not running, and the timer will not schedule graphical polls. my workaround was to call `qemu_mutex_unlock_iothread()` to unlock the mutex, and wait for the program to terminate. this might not be the right solution, but seems functional.

---

conclusions:

- the graphical event loop and all graphical primitives must be called from the main thread context
- in qemu this is not possible directly; an inefficient but functional solution uses a timer programmed to call a function every few milliseconds, to poll the event loop. (if the qemu I/O event loop is executed on the main thread, the timer function will be called from the main thread context)
- all graphical operations initiated by the all other thread (like the emulator thread), must be deferred (in SDL using user events) to the graphical event loop
- a function to run graphical destruction must be registered with atexit()
- only the main thread can call exit(), to ensure the destruction functions are not called from other threads
- exiting from other threads must be also deferred to the main thread, possibly with unlocking a mutex.


I hope that helps,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Sat, 12/10 18:16, Liviu Ionescu wrote:
> conclusions:
> 
> - the graphical event loop and all graphical primitives must be called from the main thread context
> - in qemu this is not possible directly; an inefficient but functional
> solution uses a timer programmed to call a function every few milliseconds, to
> poll the event loop. (if the qemu I/O event loop is executed on the main
> thread, the timer function will be called from the main thread context)

Isn't it possible to notify the main thread with an EventNotifier and process
the deferred events in its handler?

Fam

> - all graphical operations initiated by the all other thread (like the emulator thread), must be deferred (in SDL using user events) to the graphical event loop
> - a function to run graphical destruction must be registered with atexit()
> - only the main thread can call exit(), to ensure the destruction functions are not called from other threads
> - exiting from other threads must be also deferred to the main thread, possibly with unlocking a mutex.
> 
> 
> I hope that helps,
> 
> Liviu
> 
> 
> 
> 
> 
> 
> 

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 12 Dec 2016, at 05:35, Fam Zheng <famz@redhat.com> wrote:
> 
> Isn't it possible to notify the main thread with an EventNotifier and process
> the deferred events in its handler?

ideally, the SDL main loop might be re-implemented asynchronously, for example using a pipe to pass the events, and in this case the SDL would be just a regular client of the main I/O loop.

any idea how difficult would be to push any changes to SDL?


regards,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Mon, 12/12 09:18, Liviu Ionescu wrote:
> 
> > On 12 Dec 2016, at 05:35, Fam Zheng <famz@redhat.com> wrote:
> > 
> > Isn't it possible to notify the main thread with an EventNotifier and process
> > the deferred events in its handler?
> 
> ideally, the SDL main loop might be re-implemented asynchronously, for example
> using a pipe to pass the events, and in this case the SDL would be just a
> regular client of the main I/O loop.
> 
> any idea how difficult would be to push any changes to SDL?

Not sure I understand this question, could you be more specific?

Fam

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 12 Dec 2016, at 09:39, Fam Zheng <famz@redhat.com> wrote:
> 
>> any idea how difficult would be to push any changes to SDL?
> 
> Not sure I understand this question, could you be more specific?

I guess making SDL run asynchronously will need some major changes to the SDL source code, not simple patches that can be added to the QEMU build.

do you know someone who can not only implement the patches, but also push them to the mainstream SDL repository, to have them available to everyone?


regards,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Mon, 12/12 09:43, Liviu Ionescu wrote:
> 
> > On 12 Dec 2016, at 09:39, Fam Zheng <famz@redhat.com> wrote:
> > 
> >> any idea how difficult would be to push any changes to SDL?
> > 
> > Not sure I understand this question, could you be more specific?
> 
> I guess making SDL run asynchronously will need some major changes to the SDL
> source code, not simple patches that can be added to the QEMU build.
> 
> do you know someone who can not only implement the patches, but also push them
> to the mainstream SDL repository, to have them available to everyone?

Sorry I have no idea how to answer that.. Anyone can contribute since it is an
open source project, but I myself know next to nothing about SDL project.

Fam

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 12 Dec 2016, at 11:39, Fam Zheng <famz@redhat.com> wrote:
> 
> ... I myself know next to nothing about SDL project.

ok, neither do I.


do you have any comments on the current situation of the SDL integration in QEMU? any comments on the conclusions I reached after integrating SDL2 into GNU ARM Eclipse QEMU?

personally I do not like at all polling the SDL loop every 10 ms on a timer, but I currently have no other solution. I tried to move the I/O loop on a separate thread, but the Windows implementation of `poll` works only on the main thread.


regards,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Mon, 12/12 12:20, Liviu Ionescu wrote:
> 
> > On 12 Dec 2016, at 11:39, Fam Zheng <famz@redhat.com> wrote:
> > 
> > ... I myself know next to nothing about SDL project.
> 
> ok, neither do I.
> 
> 
> do you have any comments on the current situation of the SDL integration in
> QEMU? any comments on the conclusions I reached after integrating SDL2 into
> GNU ARM Eclipse QEMU?
> 
> personally I do not like at all polling the SDL loop every 10 ms on a timer,
> but I currently have no other solution. I tried to move the I/O loop on a
> separate thread, but the Windows implementation of `poll` works only on the
> main thread.

Like I said ealier, instead of "polling" SDL events in a timer, how about
registering an EventNotifier in the main thread which is notified by the vCPU
thread upon a new graphics operation? That will fix the latency issue.

Fam

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Daniel P. Berrange]

On Mon, Dec 12, 2016 at 06:36:16PM +0800, Fam Zheng wrote:
> On Mon, 12/12 12:20, Liviu Ionescu wrote:
> > 
> > > On 12 Dec 2016, at 11:39, Fam Zheng <famz@redhat.com> wrote:
> > > 
> > > ... I myself know next to nothing about SDL project.
> > 
> > ok, neither do I.
> > 
> > 
> > do you have any comments on the current situation of the SDL integration in
> > QEMU? any comments on the conclusions I reached after integrating SDL2 into
> > GNU ARM Eclipse QEMU?
> > 
> > personally I do not like at all polling the SDL loop every 10 ms on a timer,
> > but I currently have no other solution. I tried to move the I/O loop on a
> > separate thread, but the Windows implementation of `poll` works only on the
> > main thread.
> 
> Like I said ealier, instead of "polling" SDL events in a timer, how about
> registering an EventNotifier in the main thread which is notified by the vCPU
> thread upon a new graphics operation? That will fix the latency issue.

Also, note that SDL is not the only backend that requires all graphics
operations to take place in a single thread. GTK2/3 both require that
everything is done in the same thread. So if there's any parts of QEMU
that are triggering graphics operations in non-main threads, we really
need to make sure any fix applies to all QEMU graphics backends, not
just SDL2.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 12 Dec 2016, at 12:36, Fam Zheng <famz@redhat.com> wrote:
> 
> an EventNotifier in the main thread which is notified by the vCPU
> thread upon a new graphics operation? 

the idea of using notifications is ok, but I could not figure out what api to use to implement it. can you provide more details?

regards,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Mon, 12/12 13:27, Liviu Ionescu wrote:
> 
> > On 12 Dec 2016, at 12:36, Fam Zheng <famz@redhat.com> wrote:
> > 
> > an EventNotifier in the main thread which is notified by the vCPU
> > thread upon a new graphics operation? 
> 
> the idea of using notifications is ok, but I could not figure out what api to
> use to implement it. can you provide more details?

Take a look at event_notifier_set_handler() and event_notifier_set().

Or alternatively (whichever works better for you, I don't know): qemu_bh_new()
and qemu_bh_schedule().

Both can register a callback that will be called in main loop once fired.

Fam

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 12 Dec 2016, at 14:28, Fam Zheng <famz@redhat.com> wrote:
> 
> ... Take a look at event_notifier_set_handler() and event_notifier_set().
> ... qemu_bh_new() and qemu_bh_schedule().
> 
> Both can register a callback that will be called in main loop once fired.

ok, thank you, I'll consider them.

if I understand it right, this is a more portable alternative of using the SDL user events, to forward actions from the device thread to the main thread.

but this is only part of the problem, in addition to user events, graphical libraries also generate system events (like mouse, termination, etc) usually consumed by a kind of event loop. probably it is possible to set a notifier handler to run the SDL PollEvent, but, at least for SDL, I did not find any function in the public API to add a hook to trigger events associated with system events.

so, back to square one; any suggestion on how to avoid the periodic timer required to poll SDL system events?


regards,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Mon, 12/12 15:22, Liviu Ionescu wrote:
> so, back to square one; any suggestion on how to avoid the periodic timer
> required to poll SDL system events?

Good question, I've missed that!

Sadly I don't find it possible. The main thread of QEMU has to run the glib
event loop so it cannot block on SDL_WaitEvent().

Ideally SDL should provide an API to get a pollable fd (or a set of) to allow
such types of integration, not much can be done from QEMU side in its current
form.

Fam

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 12 Dec 2016, at 16:51, Fam Zheng <famz@redhat.com> wrote:
> 
> On Mon, 12/12 15:22, Liviu Ionescu wrote:
>> so, back to square one; any suggestion on how to avoid the periodic timer
>> required to poll SDL system events?
> 
> Good question, I've missed that!
> 
> Sadly I don't find it possible. The main thread of QEMU has to run the glib
> event loop so it cannot block on SDL_WaitEvent().

why should the glib event loop run as the main thread? according to my tests, on linux and macos the glib event loop can run very well on a regular thread.

the problem that I encountered with this approach was on windows, where there is no `poll()`, and the existing implementation of it in qemu does not work if moved on a different thread. I do not know if the qemu implementation is not correct, or the win32 api does not allow this.

> ... it cannot block on SDL_WaitEvent().

initially I considered that rearranging threads to free the main thread for a dedicated SDL loop that blocks on SDL_WaitEvent() is a good solution, but then I took a look at how SDL_WaitEvent() is implemented: a loop that sleeps 10 ms and checks if there are events in the queue. :-(

so the best solution would be to make SDL asynchronous and run it on the glib event loop.

> Ideally SDL should provide an API to get a pollable fd (or a set of) to allow
> such types of integration, not much can be done from QEMU side in its current
> form.

I'm glad you reached more or less the same conclusion as I mentioned in one of the previous messages:

> I guess making SDL run asynchronously will need some major changes to the SDL
> source code, not simple patches that can be added to the QEMU build.


regards,

Liviu

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Fam Zheng]

On Mon, 12/12 17:07, Liviu Ionescu wrote:
> 
> > On 12 Dec 2016, at 16:51, Fam Zheng <famz@redhat.com> wrote:
> > 
> > On Mon, 12/12 15:22, Liviu Ionescu wrote:
> >> so, back to square one; any suggestion on how to avoid the periodic timer
> >> required to poll SDL system events?
> > 
> > Good question, I've missed that!
> > 
> > Sadly I don't find it possible. The main thread of QEMU has to run the glib
> > event loop so it cannot block on SDL_WaitEvent().
> 
> why should the glib event loop run as the main thread? according to my tests,
> on linux and macos the glib event loop can run very well on a regular thread.

According to the API documentation I think it is supposed to work, but it will
be an unusual use case anyway so it's presumably less tested and I remember
hearing about things break here and there (cannot recall any detail :().

> 
> the problem that I encountered with this approach was on windows, where there
> is no `poll()`, and the existing implementation of it in qemu does not work if
> moved on a different thread. I do not know if the qemu implementation is not
> correct, or the win32 api does not allow this.
> 
> > ... it cannot block on SDL_WaitEvent().
> 
> initially I considered that rearranging threads to free the main thread for a
> dedicated SDL loop that blocks on SDL_WaitEvent() is a good solution, but then
> I took a look at how SDL_WaitEvent() is implemented: a loop that sleeps 10 ms
> and checks if there are events in the queue. :-(
> 
> so the best solution would be to make SDL asynchronous and run it on the glib
> event loop.

Maybe, but that would be a big project.

Given how SDL_WaitEvent() works as you said above, the easies solution would be
using qemu_bh_schedule_idle in the QEMU main loop and poll event there
periodically.

Fam

Re: [Qemu-devel] experience with SDL2, event loop & main thread

[Liviu Ionescu]

> On 13 Dec 2016, at 09:04, Fam Zheng <famz@redhat.com> wrote:
> 
> using qemu_bh_schedule_idle in the QEMU main loop and poll event there
> periodically.

ok, so my TODO list includes two items: to move user events on an event notifier and to run the SDL_PollEvent() on qemu_bh_schedule_idle. I'll do this, but after I'll be ready with my new release (planned by the end of this year).


regards,

Liviu