[Qemu-devel] Support for using TCG frontend as a library

[Qemu-devel] Support for using TCG frontend as a library

[Alessandro Di Federico]

Hi all,
  QEMU is a great emulator, but in recent years it has also been used
for instrumentation purposes [QIRA,AFL] or as a lifter for static
analysis purposes [rev.ng,angr,libqemu,S²E]. I'd like to hear your
take on the second use case, and the possibility of offering upstream
support for it.

The general idea is to introduce a new build configuration which
produces a library for each supported input ISA exposing the TCG
frontend in a unified way. We could call it libtcg-$ARCH.so. In
practice, given a buffer containing code for a certain architecture,
the user program loads the appropriate version of this library and asks
it to produce the corresponding TCG instructions.

I've been investigating the needs of the various projects that might be
interested in using it and they sum up to the following:

* Be able to load in the same process multiple libtcg-$ARCH.so for
  different architectures.
* Obtain the TCG instructions from code in a memory buffer.
* Dump the assembly code of the code in a memory buffer.
* Dump the TCG instructions in textual form.

For what concerns helpers, it would be nice to have some metadata about
them, for instance the parts of the CPU state they can change. It would
also be nice to have a build configuration which produces a library
containing all the helpers ready to be used, or, even better, a library
as LLVM bitcode, which can then be further processed/analyzed.

Here you can find some relevant parts of my draft implementation part
of rev.ng:

* The interface exposed to users:
  https://polimicg.org/gitlab/revng/qemu/blob/develop/linux-user/ptc.h
* Implementation of the interface functions:
  https://polimicg.org/gitlab/revng/qemu/blob/develop/linux-user/ptc.c
* For the changes introduced elsewhere look for CONFIG_LIBTINYCODE:
  https://polimicg.org/gitlab/search?utf8=%E2%9C%93&search=CONFIG_LIBTINYCODE&group_id=&project_id=83&search_code=true&repository_ref=develop

It's rough but it works (see [rev.ng]). I'm interested to hear your
opinion and willingness to take patches. Being able to unify the
various efforts in this direction would be good, having upstream
support would be amazing.

--
Alessandro Di Federico
PhD student at Politecnico di Milano

[QIRA] http://qira.me/
[AFL] http://lcamtuf.coredump.cx/afl/ (for the black-box mode)
[rev.ng] https://rev.ng/
[angr] http://angr.io/ (currently using VEX IR, QEMU support planned)
[libqemu] https://github.com/zaddach/libqemu
[S²E] http://s2e.epfl.ch/

Re: [Qemu-devel] Support for using TCG frontend as a library

[Paolo Bonzini]

On 27/11/2016 20:32, Alessandro Di Federico wrote:
> 
> I've been investigating the needs of the various projects that might be
> interested in using it and they sum up to the following:
> 
> * Be able to load in the same process multiple libtcg-$ARCH.so for
>   different architectures.
> * Obtain the TCG instructions from code in a memory buffer.
> * Dump the assembly code of the code in a memory buffer.
> * Dump the TCG instructions in textual form.
> 
> For what concerns helpers, it would be nice to have some metadata about
> them, for instance the parts of the CPU state they can change. It would
> also be nice to have a build configuration which produces a library
> containing all the helpers ready to be used, or, even better, a library
> as LLVM bitcode, which can then be further processed/analyzed.
> 
> Here you can find some relevant parts of my draft implementation part
> of rev.ng:
> 
> * The interface exposed to users:
>   https://polimicg.org/gitlab/revng/qemu/blob/develop/linux-user/ptc.h
> * Implementation of the interface functions:
>   https://polimicg.org/gitlab/revng/qemu/blob/develop/linux-user/ptc.c
> * For the changes introduced elsewhere look for CONFIG_LIBTINYCODE:
>   https://polimicg.org/gitlab/search?utf8=%E2%9C%93&search=CONFIG_LIBTINYCODE&group_id=&project_id=83&search_code=true&repository_ref=develop

It's pretty clean!  I would rather avoid the duplicate enums, possibly
by automatically generating large parts of ptc.h, but that's pretty much
it.  I see that you check that the constants match (that cpp stuff is
disgusting :)), and that is already a good thing.

As to dumping the assembly code, I think that's beyond the scope of
QEMU---rather, if there is an existing library to do so, QEMU would like
to use it because currently we're stuck with an old GPLv2-compatible
version of GNU binutils.  For everything else it makes sense to use a
"librarified" QEMU frontend and even backend.

It would even be better, I think, to make linux-user a client of the
library itself, because this will prevent bitrot.

As to the helpers, QEMU has annotations of helpers that read globals,
write globals or have side effects.  However, they are pretty new so
they aren't used too much.  In general we prefer new helpers to be pure,
but this is not true of much legacy code that dates back to even before
TCG (i.e. ~2008).

Thanks,

Paolo

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 29 Nov 2016, at 18:26, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> ... It's pretty clean!  ...

if you manage to make the TCG frontend as a library, it would be great to check that it can also be used from C++.

the current QEMU headers prevent this, by using C++ reserved words (like `class`), which is a real pity.


regards,

Liviu

Re: [Qemu-devel] Support for using TCG frontend as a library

[Alessandro Di Federico]

On Tue, 29 Nov 2016 17:26:59 +0100
Paolo Bonzini <pbonzini@redhat.com> wrote:

> It's pretty clean!  I would rather avoid the duplicate enums, possibly
> by automatically generating large parts of ptc.h, but that's pretty
> much it.  I see that you check that the constants match (that cpp
> stuff is disgusting :)), and that is already a good thing.

Does QEMU already have some mechanism to generate code? Otherwise I can
try to factor out the enum in a file that can be included in multiple
places.

> As to dumping the assembly code, I think that's beyond the scope of
> QEMU---rather, if there is an existing library to do so, QEMU would
> like to use it because currently we're stuck with an old
> GPLv2-compatible version of GNU binutils.  For everything else it
> makes sense to use a "librarified" QEMU frontend and even backend.

My idea was to offer it externally, if available, more for debugging
purposes than for actual usage, in a "best-effort" way, let's say.
Isolating the backend too makes sense, but I'm not sure how much
interest there is on this, I'll first focus on exposing the frontends,
which is the killer feature for many of us.

> It would even be better, I think, to make linux-user a client of the
> library itself, because this will prevent bitrot.

This is something I didn't think about, it might be interesting indeed.
But why only linux-user and not full system emulation too?

> As to the helpers, QEMU has annotations of helpers that read globals,
> write globals or have side effects.  However, they are pretty new so
> they aren't used too much.  In general we prefer new helpers to be
> pure, but this is not true of much legacy code that dates back to
> even before TCG (i.e. ~2008).

If most how the new helpers will be pure it might make sense to make a
one-time effort to annotate existing helpers with the list of parts of
the CPU they might access. I currently have an LLVM pass which analyzes
the helpers and collect programmatically this information. This might
be used for an initial collection of such information, or maybe even
run periodically (before each release?).

However, I'd leave discussion for this things for later. In the coming
weeks I'll try to come up with some self-contained patches for review.

On Tue, 29 Nov 2016 18:55:24 +0200
Liviu Ionescu <ilg@livius.net> wrote:

> if you manage to make the TCG frontend as a library, it would be
> great to check that it can also be used from C++.
> 
> the current QEMU headers prevent this, by using C++ reserved words
> (like `class`), which is a real pity.

I'm already using it from C++ (see rev.ng)! Users only need to include
ptc.h, nothing else.

Thanks for your responses, expect patches.

--
Alessandro Di Federico
PhD student at Politecnico di Milano

Re: [Qemu-devel] Support for using TCG frontend as a library

[Paolo Bonzini]

----- Original Message -----
> From: "Alessandro Di Federico" <ale+qemu@clearmind.me>
> To: "Paolo Bonzini" <pbonzini@redhat.com>, "Liviu Ionescu" <ilg@livius.net>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>
> Sent: Thursday, December 1, 2016 12:01:12 AM
> Subject: Re: [Qemu-devel] Support for using TCG frontend as a library
> 
> On Tue, 29 Nov 2016 17:26:59 +0100
> Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > It's pretty clean!  I would rather avoid the duplicate enums, possibly
> > by automatically generating large parts of ptc.h, but that's pretty
> > much it.  I see that you check that the constants match (that cpp
> > stuff is disgusting :)), and that is already a good thing.
> 
> Does QEMU already have some mechanism to generate code? Otherwise I can
> try to factor out the enum in a file that can be included in multiple
> places.

If you count scripts/shaderinclude.pl, scripts/feature_to_c.sh and
the like, yes.  The biggest is probably tracetool.  It's ad hoc.

> > As to dumping the assembly code, I think that's beyond the scope of
> > QEMU---rather, if there is an existing library to do so, QEMU would
> > like to use it because currently we're stuck with an old
> > GPLv2-compatible version of GNU binutils.  For everything else it
> > makes sense to use a "librarified" QEMU frontend and even backend.
> 
> My idea was to offer it externally, if available, more for debugging
> purposes than for actual usage, in a "best-effort" way, let's say.
> Isolating the backend too makes sense, but I'm not sure how much
> interest there is on this, I'll first focus on exposing the frontends,
> which is the killer feature for many of us.

Yes, of course.

> > It would even be better, I think, to make linux-user a client of the
> > library itself, because this will prevent bitrot.
> 
> This is something I didn't think about, it might be interesting indeed.
> But why only linux-user and not full system emulation too?

It would simplify the library.  The front-end and helpers have some differences
between usermode and softmmu, and the latter is much more intertwined with
the rest of the QEMU infrastructure (MMU indices, limits on multi-page
translations, etc.)

> If most how the new helpers will be pure it might make sense to make a
> one-time effort to annotate existing helpers with the list of parts of
> the CPU they might access. I currently have an LLVM pass which analyzes
> the helpers and collect programmatically this information.

It can also be used to guide the conversion to pure helpers, I guess.
For example, helpers such as helper_divb_AL should definitely get and
return globals instead of reading/writing to CPUX86State (in the specific
case of division, of course they would keep the side effects because they
can raise a division by zero exception).

Paolo

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 1 Dec 2016, at 10:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
>> But why only linux-user and not full system emulation too?
> 
> It would simplify the library.  The front-end and helpers have some differences
> between usermode and softmmu, and the latter is much more intertwined with
> the rest of the QEMU infrastructure (MMU indices, limits on multi-page
> translations, etc.)

hmmm... isn't it possible to split somehow the QEMU infrastructure from the ARM TCG?

I am also very interested in simplifying GNU ARM Eclipse QEMU, for Cortex-M emulation most of the features required to support Linux are only a complication of little use.


regards,

Liviu

Re: [Qemu-devel] Support for using TCG frontend as a library

[Peter Maydell]

On 1 December 2016 at 11:54, Liviu Ionescu <ilg@livius.net> wrote:
>
>> On 1 Dec 2016, at 10:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>> But why only linux-user and not full system emulation too?
>>
>> It would simplify the library.  The front-end and helpers have some differences
>> between usermode and softmmu, and the latter is much more intertwined with
>> the rest of the QEMU infrastructure (MMU indices, limits on multi-page
>> translations, etc.)
>
> hmmm... isn't it possible to split somehow the QEMU infrastructure from the ARM TCG?

Possible, certainly. Simple? That's a different question :-)

In general I would favour trying to clean up QEMU's code so
that it is less interdependent (which would make it I think
easier to maintain and test, as well as potentially easier
to use in other contexts); but I think we need to try to tackle
that gradually and doing the easier bits of disentangling first.

thanks
-- PMM

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 1 Dec 2016, at 14:38, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> ... clean up QEMU's code so
> that it is less interdependent ...

that's a good idea anyway, but this does not address the current issue.

if I'd have a separate library with ARM TCG, for Cortex-M emulation I'd probably write a simple memory management routine, to address flash, ram & peripherals and with an equally simple variant of the peripheral implementation I'd be done. no need to handle MMU, no need to worry about VM, KVM, save/restore objects, monitor, etc (actually no need for the complicated objects implementation at all).


regards,

Liviu

Re: [Qemu-devel] Support for using TCG frontend as a library

[Peter Maydell]

On 1 December 2016 at 13:33, Liviu Ionescu <ilg@livius.net> wrote:
>> On 1 Dec 2016, at 14:38, Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> ... clean up QEMU's code so
>> that it is less interdependent ...
>
> that's a good idea anyway, but this does not address the current issue.
>
> if I'd have a separate library with ARM TCG, for Cortex-M emulation
> I'd probably write a simple memory management routine, to address
> flash, ram & peripherals and with an equally simple variant of the
> peripheral implementation I'd be done. no need to handle MMU,
> no need to worry about VM, KVM, save/restore objects, monitor,
> etc (actually no need for the complicated objects implementation at all).

The advantage of the 'MMU' layer for M profile is that it is the
path to having a working MPU (which we don't implement quite yet
but it would be nice to some day). The system emulation also pretty
much assumes our softmmu layer as the way to handle fast path
(memory) versus slowpath (device emulation, watchpoints, etc).

The advantage of save/restore is that users can do snapshotting, which
is a really useful debug technique.
You could throw away all our implementations of devices and
backends and so on, but then you'd have to reimplement them all
yourself (including things like network device backends), which
is a fair bit of work.

thanks
-- PMM

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 1 Dec 2016, at 16:38, Peter Maydell <peter.maydell@linaro.org> wrote:

> ... network device backends ... is a fair bit of work.

yes, that's a good point, and I also plan to add Ethernet support to some of my boards, but I need first to fix some issues that I consider more important, like Cortex-M system peripherals (NVIC!), exceptions, support for M4/M7, etc.


as for snapshotting, I'm not sure how I can use this for bare metal devices (hints highly appreciated).


regards,

Liviu

Re: [Qemu-devel] Support for using TCG frontend as a library

[Peter Maydell]

On 1 December 2016 at 18:39, Liviu Ionescu <ilg@livius.net> wrote:
> as for snapshotting, I'm not sure how I can use this for bare
> metal devices (hints highly appreciated).

You need a QCOW2 disk to store the snapshots on, but it
doesn't actually need to be accessible to the guest system,
so it should work fine for embedded boards with no hard
disk. Taking savevm snapshots doesn't need any cooperation
from the guest OS, so it will work with bare-metal setups.
https://translatedcode.wordpress.com/2015/07/06/tricks-for-debugging-qemu-savevm-snapshots/
has some info on how to do this.

I don't know whether anybody's tested this with M profile:
snapshotting does need support from all the devices in
the system to save and restore their state (otherwise
after restore the system misbehaves because devices
which didn't restore their state are out of sync with
the rest of the system). It's generally fairly easy to
add support when a device is missing it, though.

thanks
-- PMM

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 1 Dec 2016, at 21:13, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> You need a QCOW2 disk to store the snapshots on, ...
> Taking savevm snapshots doesn't need any cooperation
> from the guest OS ..
> I don't know whether anybody's tested this with M profile:

does your Stellaris board support this?

assuming I fix my devices to save their status and I can correctly resume from a given snapshot, I still fail to understand how to integrate this feature in the usual debugging workflow.

the way GNU ARM Eclipse QEMU is currently used is similar to any JTAG connected physical board, the user starts a GDB client that connects to QEMU as a GDB server, writes the 'flash' content and then steps through the code. there is really no difference from a physical board. with a properly configured Eclipse, the user has nothing special to do, only select either SEGGER J-Link or QEMU, the rest is automatic.

the second use case is to run unit tests, when a script starts QEMU in semihosting mode, passes the elf file, and get an exit code and an XML with the detailed test results. once the script is functional it can be repeatedly executed by continuous integration servers, and there can be no interaction with QEMU during the tests.


what would be a debug scenario that uses snapshots?


regards,

Liviu

Re: [Qemu-devel] Support for using TCG frontend as a library

[Peter Maydell]

On 1 December 2016 at 19:45, Liviu Ionescu <ilg@livius.net> wrote:
>
>> On 1 Dec 2016, at 21:13, Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> You need a QCOW2 disk to store the snapshots on, ...
>> Taking savevm snapshots doesn't need any cooperation
>> from the guest OS ..
>> I don't know whether anybody's tested this with M profile:
>
> does your Stellaris board support this?
>
> assuming I fix my devices to save their status and I can correctly
> resume from a given snapshot, I still fail to understand how to
> integrate this feature in the usual debugging workflow.

The most useful approach is that you can set up a complicated
situation (eg "boot my embedded RTOS, start application"),
snapshot at that point, and then you can repeatedly restart
debugging from the snapshot without having to take ages going
through the bootup process each time. (Particularly useful if
you turn on heavyweight tracing which can turn bootup from
"fairly fast" to "incredibly slow".)

thanks
-- PMM

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 2 Dec 2016, at 11:40, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
>> ... integrate this feature in the usual debugging workflow.
> 
> The most useful approach is that you can set up a complicated
> situation (eg "boot my embedded RTOS, start application"),
> snapshot at that point, and then you can repeatedly restart
> debugging from the snapshot without having to take ages going
> through the bootup process each time. (Particularly useful if
> you turn on heavyweight tracing which can turn bootup from
> "fairly fast" to "incredibly slow".)

I see your point, and I'm convinced that for your use cases booting linux and starting applications is a big deal of effort.

for the bare metal use cases, "going through the bootup process" is quite lightwheight, setting a few registers and the application starts.

as for integration with the current workflow, it might be done, but it requires quite a lot of efforts, and the results are only for a very limited audience, if any.

on the other side, what would be really useful for Cortex-M, are the instruction and data tracing features available for some devices, usually available only with very expensive multi-pin JTAG probes on physical devices; were these ARM features considered?

---

to summarise, the requirements for emulating Cortex-M devices are much, much lower than for your current large targets using OSes.

as a result, I'm seriously considering simplifying the GNU ARM Eclipse QEMU setup and tapping directly into the TCG, so packing it as a library will definitely be a useful feature.


regards,

Liviu

Re: [Qemu-devel] Support for using TCG frontend as a library

[Peter Maydell]

On 2 December 2016 at 10:12, Liviu Ionescu <ilg@livius.net> wrote:
> I see your point, and I'm convinced that for your use cases booting
> linux and starting applications is a big deal of effort.
>
> for the bare metal use cases, "going through the bootup process"
> is quite lightwheight, setting a few registers and the application
> starts.

Right, but if you have a bug which requires your application to
sit there processing for half an hour (or even five minutes)
before it appears, it's nice not to spend that time.

> as for integration with the current workflow, it might be done,
> but it requires quite a lot of efforts, and the results are only
> for a very limited audience, if any.

You could surface the basic functionality with a 'snapshot'
button that invoked the monitor "savevm" command, and a
"start from snapshot" that just added '-loadvm snapshotname' to
the QEMU command line. Everything else (including connecting to
gdbstub) should just work as usual.

> on the other side, what would be really useful for Cortex-M, are
> the instruction and data tracing features available for some
> devices, usually available only with very expensive multi-pin
> JTAG probes on physical devices; were these ARM features considered?

These are painful to add to TCG (which generally doesn't bother
to keep track of information it doesn't need for execution),
and they'd make things very slow (plus they're very invasive
changes to the TCG frontend code). So I don't think they're
a bad idea but they're not at the top of my priority list at the
moment.

thanks
-- PMM

Re: [Qemu-devel] Support for using TCG frontend as a library

[Liviu Ionescu]

> On 2 Dec 2016, at 12:24, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> Right, but if you have a bug which requires your application to
> sit there processing for half an hour (or even five minutes)
> before it appears, it's nice not to spend that time.

fully agree. for physical targets, the GNU ARM Eclipse debugging plug-ins have a feature called "connect to running target", when the debugger attaches to a live target without any reset or flash reprogramming.

> You could surface the basic functionality with a 'snapshot'
> button that invoked the monitor "savevm" command, and a
> "start from snapshot" that just added '-loadvm snapshotname' to
> the QEMU command line. Everything else (including connecting to
> gdbstub) should just work as usual.

well, yes, something like this can be done, but it is not that simple, normally the debugging plug-ins will reset the board and reprogram the flash, but a similar approach to "connect to running target" can be used.

but besides the gui aspects, the deeper conclusion from this discussion is that, regardless the implementation (with the current QEMU very complicated objects, or with the C++ natural objects that I'd like to use), the entire tree of peripherals should be fully serialisable, otherwise the peripheral status cannot be restored correctly.

I'll consider this and possibly adjust my current implementation, which does not use links consistently.

do you have any other suggestions for making Cortex-M devices 'snaphotable'? (since I understand the ones used in the Stellaris implementation are not).

> 
>> on the other side, what would be really useful for Cortex-M, are
>> the instruction and data tracing features available for some
>> devices, usually available only with very expensive multi-pin
>> JTAG probes on physical devices; were these ARM features considered?
> 
> These are painful to add to TCG (which generally doesn't bother
> to keep track of information it doesn't need for execution),
> and they'd make things very slow

any estimate on how much slower? anyway I think this should be a runtime option, enabled only when really needed; getting the trace output is also expensive on physical targets, so, I think a reasonable speed penalty can be accepted, at least for most Cortex-M devices, which normally run at some tens of MHz.

> (plus they're very invasive
> changes to the TCG frontend code). So I don't think they're
> a bad idea but they're not at the top of my priority list at the
> moment.

I see. in case you'll reconsider, please let me know, maybe I can help.


regards,

Liviu