Re: [Qemu-devel] [PATCH 1/2] coroutine: introduce coroutines

[Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>]

On Tue, May 24, 2011 at 08:37:50PM +0100, Jamie Lokier wrote:
> Daniel P. Berrange wrote:
> > On Wed, May 11, 2011 at 03:45:39PM +0200, Paolo Bonzini wrote:
> > > On 05/11/2011 03:05 PM, Anthony Liguori wrote:
> > > >>
> > > >>A very slow way, too (on Windows at least if you use qemu_cond...).
> > > >
> > > >That doesn't mean you can't do a fiber implementation for Windows... but
> > > >having a highly portable fallback is a good thing.
> > > 
> > > I agree but where would you place it, since QEMU is only portable to
> > > POSIX and Windows?
> > > 
> > > osdep-$(CONFIG_POSIX) += coroutine-posix.c
> > > osdep-$(CONFIG_WIN32) += coroutine-win32.c
> > > osdep-??? += coroutine-fallback.c
> > 
> > NetBSD forbids the use of 'makecontext' in any application
> > which also links to libpthread.so[1]. We used makecontext in
> > GTK-VNC's coroutines and got random crashes in threaded
> > apps running on NetBSD. So for NetBSD we tell people to use
> > the thread based coroutines instead.
> 
> You have to use swapcontext(), no wait, you have to use setjmp(), no wait,
> _setjmp(), no wait, threads.... Read on.
> 
> From Glibc's FAQ, setjmp/longjmp are not portable choices:
> 
>     - UNIX provides no other (portable) way of effecting a synchronous
>       context switch (also known as co-routine switch).  Some versions
>       support this via setjmp()/longjmp() but this does not work
>       universally.
> 
> So in principle you should use swapcontext() in portable code.
> 
> (By the way, Glibc goes on about how it won't support swapcontext()
> from async signal handlers, i.e. preemption, on some architectures
> (IA-64/S-390), and I know it has been very subtly broken from a signal
> handler on ARM.  Fair enough, somehow disappointing, but doesn't
> matter for QEMU coroutines.)
> 
> But swapcontext() etc. have been withdrawn from POSIX 2008:
> 
>     - Functions to be deleted
> 
>       Legacy: Delete all legacy functions except utimes (which should not be legacy).
>       OB: Default position is to delete all OB functions.
> 
>       XSI Functions to change state
> 
>       ....
>       _setjmp and _longjmp. Should become obsolete.
>       ....
>       getcontext, setcontext, makecontext and swapcontext are already
>       marked OB and should be withdrawn. And header file <ucontext.h>. 
> 
> OB means obsolescent.  They were marked obsolescent a few versions
> prior, with the rationale that you can use threads instead...

Yep, aware of this but at the end of the day these functions are
commonly available.

> It's not surprising that NetBSD forbids makecontext() with
> libpthread.so.  I suspect old versions of FreeBSD, OpenBSD, DragonFly
> BSD, (and Mac OS X?), have the same restriction, because they have a
> similar pthreads evolutionary history to LinuxThreads.  LinuxThreads
> also breaks when using coroutines that switch stacks, because it uses
> the stack pointer to know the current thread.
> 
> (LinuxThreads is old now, but that particular quirk still affects me
> because some uCLinux platforms, on which I wish to use coroutines, still
> don't have working NPTL - but they aren't likely to be running QEMU :-)

That is nasty.

> Finally, if you are using setjmp/longjmp, consider (from FreeBSD man page):
> 
>     The setjmp()/longjmp() pairs save and restore the signal mask
>     while _setjmp()/_longjmp() pairs save and restore only the
>     register set and the stack.  (See sigprocmask(2).)
> 
> As setjmp/longjmp were chosen for performance, you may wish to use
> _setjmp/_longjmp instead (when available), as swizzling the signal
> mask on each switch may involve a system call and be rather slow.

Thanks, I read about that but didn't try to implement special cases
because I don't have relevant OSes here to test against.

My current plan is to try using sigaltstack(2) instead of
makecontext()/swapcontext() as a hack since OpenBSD doesn't have
makecontext()/swapcontext().

TBH I'm almost at the stage where I think we should just use threads
and/or async callbacks, as appropriate.  Hopefully I'll be able to cook
up a reasonably portable implementation of coroutines though, because
the prospect of having to go fully threaded or do async callbacks isn't
attractive in many cases.

Stefan