Re: [Qemu-devel] Improve QEMU performance with LLVM codegen and other techniques

[陳韋任 <chenwj@iis.sinica.edu.tw>]

Hi Alex,

> Very cool! I was thinking about this for a while myself now. It's especially appealing these days since you can do the hotspot optimization in a separate thread :).
> 
> Especially in system mode, you also need to flush when tb_flush() is called though. And you have to make sure to match hflags and segment descriptors for the links - otherwise you might end up connecting TBs from different processes :).

  I'll check the tb_flush again. IIRC, we make the code cache big enough so that
there is no need to flush the code cache. But I think we still need to deal with
it in the end.

  The block linking is done by QEMU and we leave it alone. But I don't know QEMU
ever does hflags and segment descriptors check before doing block linking. Could
you point it out? Anyway, here is how we form trace from a set of basic blocks.

1. We insert instrumented code at the beginning of each TCG block to collect how
   many times this block being executed.

2. When a block's execution time, say block A, reaches a pre-defined threshold,
   we follow the run time execution path to collect block B followed A and so on
   to form a trace. This approach is called NET (Next-Executing Tail) [1].

3. Then a trace composed of TCG blocks is sent to a LLVM translator. The translator
   generates the host binary for the trace into a LLVM code cache, and patch the
   beginning of block A (in QEMU code cache) so that anyone executing block A will 
   jump to the corresponding trace and execute.

Above is block to trace link. I think there is no need to do hflags and segment
descriptors check, right? Although I set the trace length to one basic block at
the moment (make the situation simpler), I think we still don't have to check
the blocks' hflags and segment descriptors in the trace to see if they match.
 
> > successfully, then login and run some benchmark on it. As a very first step, we
> > make a very high threshold on trace building. In other words, a basic block must
> > be executed *many* time to trigger the trace building process. Then we lower the
> > threshold a bit at a time to see how things work. When something goes wrong, we
> > might get kernel panic or the system hangs at some point on the booting process.
> > I have no idea on how to solve this kind of problem. So I'd like to seek for
> > help/experience/suggestion on the mailing list. I just hope I make the whole
> > situation clear to you. 
> 
> I don't see any better approach to debugging this than the one you're already taking. Try to run as many workloads as you can and see if they break :). Oh and always make the optimization optional, so that you can narrow it down to it and know you didn't hit a generic QEMU bug.

  You mean make the trace optimization optional? We have tested our framework in
LLVM-only mode. which means we replace TCG with LLVM entirely. It's _very_ slow
but works. What the generic QEMU bug is? We use QEMU 0.13 and just rely on its
emulation part right now. Does recent version fix major bugs in the emulation
engine?

  Thanks for your advices. :-)

[1] http://www.cs.virginia.edu/kim/docs/micro05.pdf

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj