[Qemu-devel] forking a virtual machine

[Tim Newsham <tim.newsham@gmail.com>]

Hi,  We've got an unusual use case for our qemu virtual machines.
We're using a qemu-based system to run a bunch of instrumented
test cases. QEMU provides the instrumented execution environment.
Before we start testing we setup the test environment, which includes
booting an operating system as a guest.  Once everything is ready, we
fork the qemu process once for each test case, with the parent controlling
the test operation while the child performs a test.  (For more details,
see our blog post:
https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/june/project-triforce-run-afl-on-everything/,
full code
can be found at https://github.com/nccgroup/TriforceAFL)

Right now our fork process is very hacky and gross, because linux
fork() intentionally does not handle forking a multithreaded program.
 We have a special
cpu instruction which triggers the forking to go into test mode.  It sets a
flag, which causes the one and only cpu thread to exit.  Right before it
exits, it sends a signal to the iothread by writing over a pipe.  The
iothread
gets woken up and starts controlling test cases by forking children.
After each fork, it starts up a new cpu to replace the one that exited
before the fork.

This is less than ideal for several reasons.  Most importantly for us, we
would like to be able to communicate informatoin about new JITs that
occur from the child back to the parent so that the parent can reproduce
the JIT and save future children from having to keep JITting the same
thing.  This doesnt currently work (I think because the cpu has exited,
and there is no cpu left in the parent process).

We would also like to have more elegant forking code.

And so my question here -- Is there a better solution that we could use
to support forking a qemu VM?  Are there any single-threaded variants of
qemu that would work well here?  Are there any interesting tricks to pause
and resume a cpu across forks in a clean fashion?  Has anyone else
already worked on this problem?

-- 
Tim Newsham | www.thenewsh.com/~newsham | @newshtwit | thenewsh.blogspot.com