[Qemu-devel] QEMU to generate host binary

[Qemu-devel] QEMU to generate host binary

[Ayaz Akram]

[-- Attachment #1: Type: text/plain, Size: 231 bytes --]

Hello !
Is anyone aware of an effort to produce an executable binary for host using
qemu. I mean is it possible that qemu generate a binary for whatever
application it is emulating, which can later be run directly on host?

Thanks

[-- Attachment #2: Type: text/html, Size: 293 bytes --]

Re: [Qemu-devel] QEMU to generate host binary

[Peter Crosthwaite]

On Fri, Jun 26, 2015 at 12:33 PM, Ayaz Akram <aaqemu@gmail.com> wrote:
> Hello !
> Is anyone aware of an effort to produce an executable binary for host using
> qemu. I mean is it possible that qemu generate a binary for whatever
> application it is emulating, which can later be run directly on host?
>

I'm not sure what this binary would mean just yet. Are you extracting
just the guest + its runtime state to a binary that picks up where the
guest left off?

Or are you including the machine emulator (i.e. QEMU itself) in this
new binary to avoid having to load the guest it while picking up where
left off?

Regards,
Peter

> Thanks
>

Re: [Qemu-devel] QEMU to generate host binary

[Ayaz Akram]

[-- Attachment #1: Type: text/plain, Size: 1081 bytes --]

> Let's say qemu is running in System Emulation Mode, when it runs guest's
> binary, it can log the translated code for host. Is it possible to merge
> that translated code and other sections of guest's binary to make a binary
> which can be run directly on host.
>
> Thanks
>
> On Fri, Jun 26, 2015 at 11:34 PM, Peter Crosthwaite <
> peter.crosthwaite@xilinx.com> wrote:
>
>> On Fri, Jun 26, 2015 at 12:33 PM, Ayaz Akram <aaqemu@gmail.com> wrote:
>> > Hello !
>> > Is anyone aware of an effort to produce an executable binary for host
>> using
>> > qemu. I mean is it possible that qemu generate a binary for whatever
>> > application it is emulating, which can later be run directly on host?
>> >
>>
>> I'm not sure what this binary would mean just yet. Are you extracting
>> just the guest + its runtime state to a binary that picks up where the
>> guest left off?
>>
>> Or are you including the machine emulator (i.e. QEMU itself) in this
>> new binary to avoid having to load the guest it while picking up where
>> left off?
>>
>> Regards,
>> Peter
>>
>> > Thanks
>> >
>>
>
>

[-- Attachment #2: Type: text/html, Size: 1797 bytes --]

Re: [Qemu-devel] QEMU to generate host binary

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote:
> > Let's say qemu is running in System Emulation Mode, when it runs guest's
> > binary, it can log the translated code for host. Is it possible to merge
> > that translated code and other sections of guest's binary to make a binary
> > which can be run directly on host.

No, because of self-modifying code, run-time code loading, etc.

It is not possible to statically translate an executable (in the general
case).

There are architectures where it is possible due to restrictions (e.g.
no code loading, all jump destinations are known in advance, etc) but
the popular x86, ARM, etc architectures allow too much freedom to be
amenable to static translation.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] QEMU to generate host binary

[Peter Crosthwaite]

On Mon, Jun 29, 2015 at 8:13 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote:
>> > Let's say qemu is running in System Emulation Mode, when it runs guest's

System emulation makes the problem even harder, as a system mode
binary (usually an OS or some sort) will have difficult porting from
one CPU-types system arch to another.

This is more realistic (but still very difficult and not generally
solvable) in user-mode emulation.

>> > binary, it can log the translated code for host. Is it possible to merge
>> > that translated code and other sections of guest's binary to make a binary
>> > which can be run directly on host.
>
> No, because of self-modifying code, run-time code loading, etc.
>

Ruling these two out for the moment ...

> It is not possible to statically translate an executable (in the general
> case).
>
> There are architectures where it is possible due to restrictions (e.g.
> no code loading, all jump destinations are known in advance, etc) but

Debug info with function information might give you a crude
approximation of jump targets coming from fn pointers. That + the
statically determinable jump targets might give you something for apps
that don't do anything wierd.

I'm wondering if the jump problem can be crudely solved by a fully
single-step translation. The result binary would be huge an
inefficient. But could you keep two translations around? One that uses
the statically determinable "best guess" of the jump dest table I
describe above, and a second defensive translation of the entire app
in single-step?

There are more complications however. Another one I can think of is
instructions that change runtime state and affect (re)translation
(e.g. the arm setend instruction which switches CPU endianness).

Regards,
Peter

> the popular x86, ARM, etc architectures allow too much freedom to be
> amenable to static translation.
>
> Stefan

Re: [Qemu-devel] QEMU to generate host binary

[Ayaz Akram]

[-- Attachment #1: Type: text/plain, Size: 2614 bytes --]

Thanks for your answers. The thing that i still do not get is once we have
host assembly code (output assembly log generated for compiled TBs ),
should we still worry about problems like self modifying code and other
dynamic conditions? Moreover, assuming static linking, will not this code
be enough to generate .text section of an executable that could be run
directly on host (if somehow other sections of that host executable can be
generated, which is itself difficult) ?



On Mon, Jun 29, 2015 at 1:04 PM, Peter Crosthwaite <
peter.crosthwaite@xilinx.com> wrote:

> On Mon, Jun 29, 2015 at 8:13 AM, Stefan Hajnoczi <stefanha@gmail.com>
> wrote:
> > On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote:
> >> > Let's say qemu is running in System Emulation Mode, when it runs
> guest's
>
> System emulation makes the problem even harder, as a system mode
> binary (usually an OS or some sort) will have difficult porting from
> one CPU-types system arch to another.
>
> This is more realistic (but still very difficult and not generally
> solvable) in user-mode emulation.
>
> >> > binary, it can log the translated code for host. Is it possible to
> merge
> >> > that translated code and other sections of guest's binary to make a
> binary
> >> > which can be run directly on host.
> >
> > No, because of self-modifying code, run-time code loading, etc.
> >
>
> Ruling these two out for the moment ...
>
> > It is not possible to statically translate an executable (in the general
> > case).
> >
> > There are architectures where it is possible due to restrictions (e.g.
> > no code loading, all jump destinations are known in advance, etc) but
>
> Debug info with function information might give you a crude
> approximation of jump targets coming from fn pointers. That + the
> statically determinable jump targets might give you something for apps
> that don't do anything wierd.
>
> I'm wondering if the jump problem can be crudely solved by a fully
> single-step translation. The result binary would be huge an
> inefficient. But could you keep two translations around? One that uses
> the statically determinable "best guess" of the jump dest table I
> describe above, and a second defensive translation of the entire app
> in single-step?
>
> There are more complications however. Another one I can think of is
> instructions that change runtime state and affect (re)translation
> (e.g. the arm setend instruction which switches CPU endianness).
>
> Regards,
> Peter
>
> > the popular x86, ARM, etc architectures allow too much freedom to be
> > amenable to static translation.
> >
> > Stefan
>

[-- Attachment #2: Type: text/html, Size: 3411 bytes --]

Re: [Qemu-devel] QEMU to generate host binary

[Dr. David Alan Gilbert]

* Ayaz Akram (aaqemu@gmail.com) wrote:
> Thanks for your answers. The thing that i still do not get is once we have
> host assembly code (output assembly log generated for compiled TBs ),
> should we still worry about problems like self modifying code and other
> dynamic conditions? Moreover, assuming static linking, will not this code
> be enough to generate .text section of an executable that could be run
> directly on host (if somehow other sections of that host executable can be
> generated, which is itself difficult) ?

It can certainly help, but you can still walk into a piece of code
that you've not previously translated, and a shared library for example
might change, and your code might change due to other things such as
randomisation of library loading addresses.

Various emulators in the past have dumped some part of the translation state
to disk in an attempt to save on costs when they next emulate the code;
others have tried to do background analysis to optimise the translated
code into a pretranslated blob - but they can never guarantee to get
the whole set of code translated; and that's even for a user-mode emulation
where at least you know which binary file you're trying to emulate.

In system emulation it's much more difficult because the emulator doesn't
have a view of which executable the page it's currently emulating corresponds
to; indeed that page could be shared between multiple different executables,
or in the opposite direction is often patched during runtime linking.

You say 'should we still worry about problems like self modifying code and other
dynamic conditions?' - well you hope that for most 'normal' programs that
self modification doesn't happen; but it keeps turning up even when you don't
expect it, e.g. loading/unloading of plugins or if you happen to be
emulating a JIT (such as qemu!).  And QEMU (especially in system mode)
doesn't know whether the binary that's being run is a nice safe friendly
static binary or something that's going to do some mad optimisation itself.

One example of the type of thing you can do is save statistics information
about a piece of executable, e.g. 'this code is hot' so you know it's
a hot loop that's worth optimising; that can pay off if you have multiple
levels of interpreter/optimisation in your emulator.

Dave

> 
> 
> 
> On Mon, Jun 29, 2015 at 1:04 PM, Peter Crosthwaite <
> peter.crosthwaite@xilinx.com> wrote:
> 
> > On Mon, Jun 29, 2015 at 8:13 AM, Stefan Hajnoczi <stefanha@gmail.com>
> > wrote:
> > > On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote:
> > >> > Let's say qemu is running in System Emulation Mode, when it runs
> > guest's
> >
> > System emulation makes the problem even harder, as a system mode
> > binary (usually an OS or some sort) will have difficult porting from
> > one CPU-types system arch to another.
> >
> > This is more realistic (but still very difficult and not generally
> > solvable) in user-mode emulation.
> >
> > >> > binary, it can log the translated code for host. Is it possible to
> > merge
> > >> > that translated code and other sections of guest's binary to make a
> > binary
> > >> > which can be run directly on host.
> > >
> > > No, because of self-modifying code, run-time code loading, etc.
> > >
> >
> > Ruling these two out for the moment ...
> >
> > > It is not possible to statically translate an executable (in the general
> > > case).
> > >
> > > There are architectures where it is possible due to restrictions (e.g.
> > > no code loading, all jump destinations are known in advance, etc) but
> >
> > Debug info with function information might give you a crude
> > approximation of jump targets coming from fn pointers. That + the
> > statically determinable jump targets might give you something for apps
> > that don't do anything wierd.
> >
> > I'm wondering if the jump problem can be crudely solved by a fully
> > single-step translation. The result binary would be huge an
> > inefficient. But could you keep two translations around? One that uses
> > the statically determinable "best guess" of the jump dest table I
> > describe above, and a second defensive translation of the entire app
> > in single-step?
> >
> > There are more complications however. Another one I can think of is
> > instructions that change runtime state and affect (re)translation
> > (e.g. the arm setend instruction which switches CPU endianness).
> >
> > Regards,
> > Peter
> >
> > > the popular x86, ARM, etc architectures allow too much freedom to be
> > > amenable to static translation.
> > >
> > > Stefan
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] QEMU to generate host binary

[Alex Bennée]

Dr. David Alan Gilbert <dgilbert@redhat.com> writes:

> * Ayaz Akram (aaqemu@gmail.com) wrote:
>> Thanks for your answers. The thing that i still do not get is once we have
>> host assembly code (output assembly log generated for compiled TBs ),
>> should we still worry about problems like self modifying code and other
>> dynamic conditions? Moreover, assuming static linking, will not this code
>> be enough to generate .text section of an executable that could be run
>> directly on host (if somehow other sections of that host executable can be
>> generated, which is itself difficult) ?
>
> It can certainly help, but you can still walk into a piece of code
> that you've not previously translated, and a shared library for example
> might change, and your code might change due to other things such as
> randomisation of library loading addresses.
<snip>
> You say 'should we still worry about problems like self modifying code and other
> dynamic conditions?' - well you hope that for most 'normal' programs that
> self modification doesn't happen; but it keeps turning up even when you don't
> expect it, e.g. loading/unloading of plugins or if you happen to be
> emulating a JIT (such as qemu!).

I think PLT patching is another case of self modifying code you often
come across.

-- 
Alex Bennée