[Qemu-devel] Endian and userspace issues

[Qemu-devel] Endian and userspace issues

[anarkhos]

Hello,
	I've read the technical documentation, but there are some issues left unanswered in my head. I have very little experience with the x86 ISA so please bear with me if I missed something obvious.

	The short version of this email is: 

Has anyone considered emulating little endian CPUs on big endian platforms by emulating a big endian version of the CPU? I believe VirtualPC 5 does this (not sure about 6). Obviously there is a speed advantage in doing this, but the other advantage lies in the ability for non-native little endian executables to call big endian libraries without having to swap bytes.

	I became interested in QEMU when a Darwin port was revealed. Unfortunately, user mode emulation isn't supported yet. However, even when it is, I don't think (as I understand it) it will allow non-native binaries (in either ELF or Mach-O format) to call native ones. I found it interesting the documentation touts that user mode emulation can run WINE, but the entire WINE set of libs would have to run under emulation.

	I understand that there is an inherent difficulty in that x86 executables assume they are running in little endian mode (I call it mode since some CPUs can run in either), but if one wants to have a shared user space with one set of natively optimized libraries what better way to implement it? We would have faster linking and faster CPU emulation.

Re: [Qemu-devel] Endian and userspace issues

[Paul Brook]

On Tuesday 04 January 2005 19:44, anarkhos@vfemail.net wrote:
>  I became interested in QEMU when a Darwin port was revealed.
> Unfortunately, user mode emulation isn't supported yet. However, even when
> it is, I don't think (as I understand it) it will allow non-native binaries
> (in either ELF or Mach-O format) to call native ones. I found it
> interesting the documentation touts that user mode emulation can run WINE,
> but the entire WINE set of libs would have to run under emulation.
>
>  I understand that there is an inherent difficulty in that x86 executables
> assume they are running in little endian mode (I call it mode since some
> CPUs can run in either), but if one wants to have a shared user space with
> one set of natively optimized libraries what better way to implement it? We
> would have faster linking and faster CPU emulation.

The problem is that to mix any two different types of code (big/little endian, 
native ppc vs emultated x86, whatever) you need a well defined interface 
between the two so that you can insert thunks. These thunks do whatever 
conversion is necessary. To do this you need to know all information passed 
across the interface. In practice this means not just the actual function 
arguments, but also any data passed/returned indirectly via pointers, and any 
data accessed via global variables. 

For userspace emulation the thunked interface is the linux syscall layer. This 
is designed to be a clean interface between two different types of code, so 
translating from guest syscalls to host syscalls is relatively simple.

However shared libraries tend to have much less cleanly defined interfaces. 
They tend do share data structures, and be much more closely linked. This 
makes adding the translation layer between the two much more difficult, if 
not impossible. It generally requires designing the interface with this in 
mind from the start, and in general can't be retrofitted to existing 
libraries. Shared libraries (aka dlls) share an address space with the main 
application, so tend to be very hard to disentangle from each other.

Paul

Re: [Qemu-devel] Endian and userspace issues

[anarkhos]

>
>However shared libraries tend to have much less cleanly defined interfaces.
>They tend do share data structures, and be much more closely linked. This
>makes adding the translation layer between the two much more difficult, if
>not impossible. It generally requires designing the interface with this in
>mind from the start, and in general can't be retrofitted to existing
>libraries. Shared libraries (aka dlls) share an address space with the main
>application, so tend to be very hard to disentangle from each other.
>
>Paul

I can see your point in that if a non-native app passed an address to a space of little endian memory to a native DLL which required big-endian memory...but my suggestion was that everything would be native-endian after one initial translation.

PS: WINE can insert thunk layers between the app and DLLs or between the DLLs. It currently does this for debugging purposes (printing arguments and return values), but could it not convert these values before passing them on?

Re: [Qemu-devel] Endian and userspace issues

[Paul Brook]

On Tuesday 04 January 2005 20:37, anarkhos@vfemail.net wrote:
> >However shared libraries tend to have much less cleanly defined
> > interfaces. They tend do share data structures, and be much more closely
> > linked. This makes adding the translation layer between the two much more
> > difficult, if not impossible. It generally requires designing the
> > interface with this in mind from the start, and in general can't be
> > retrofitted to existing libraries. Shared libraries (aka dlls) share an
> > address space with the main application, so tend to be very hard to
> > disentangle from each other.
> >
> >Paul
>
> I can see your point in that if a non-native app passed an address to a
> space of little endian memory to a native DLL which required big-endian
> memory...but my suggestion was that everything would be native-endian after
> one initial translation.

This only works in trivial cases. For instance anything involving 64-bit 
arithmetic on a 32-bit machine will load a 64-bit value with two separate 
32-bit loads. After your translation these will be the wrong way round.

> PS: WINE can insert thunk layers between the app and DLLs or between the
> DLLs. It currently does this for debugging purposes (printing arguments and
> return values), but could it not convert these values before passing them
> on?

But WINE knows the details of the libraries it is thunking, and can only do it 
for specific dlls, right?

Translating function arguments requires source level knowledge if the library 
(how many and what type of arguments to expect). If you only have simple 
arguments/return values this is ok, when pointers and structures are involved 
it becomes much more complicated because you need to know the purpose of each 
field.

This is reasonably straightforward if implemented at the source level (either 
library or applicaton is annotated properly and compiled specifically for 
this). For a specific special case interface (eg. mozilla plugins), it should 
be possible with a bit of work (depends on the interface). AFAICS it is not 
possible to implement a general code that will work with arbitrary 
libraries/libraries.

Paul

Re: [Qemu-devel] Endian and userspace issues

[John Davidorff Pell]

[-- Attachment #1: Type: text/plain, Size: 2970 bytes --]

I think that part of what he is suggesting is that the code that is 
little endian be translated to Big endian before execution. This would 
make the running binary "native" in memory, and so could continue to be 
closely integrated with its linked libraries.

I'd also like to note that WINE does load binaries that are linked to a 
number of libraries with no officially documented interface but manages 
to handle things rather well. Granted they don't have to deal with 
endian issues. :-)

JP

On 4 Jan 2005, at 12:16, Paul Brook wrote:

> On Tuesday 04 January 2005 19:44, anarkhos@vfemail.net wrote:
>
>> I became interested in QEMU when a Darwin port was revealed.
>> Unfortunately, user mode emulation isn't supported yet. However, even 
>> when
>> it is, I don't think (as I understand it) it will allow non-native 
>> binaries
>> (in either ELF or Mach-O format) to call native ones. I found it
>> interesting the documentation touts that user mode emulation can run 
>> WINE,
>> but the entire WINE set of libs would have to run under emulation.
>>
>>  I understand that there is an inherent difficulty in that x86 
>> executables
>> assume they are running in little endian mode (I call it mode since 
>> some
>> CPUs can run in either), but if one wants to have a shared user space 
>> with
>> one set of natively optimized libraries what better way to implement 
>> it? We
>> would have faster linking and faster CPU emulation.
>>
> The problem is that to mix any two different types of code (big/little 
> endian, native ppc vs emultated x86, whatever) you need a well defined 
> interface between the two so that you can insert thunks. These thunks 
> do whatever conversion is necessary. To do this you need to know all 
> information passed across the interface. In practice this means not 
> just the actual function arguments, but also any data passed/returned 
> indirectly via pointers, and any data accessed via global variables. 
> For userspace emulation the thunked interface is the linux syscall 
> layer. This is designed to be a clean interface between two different 
> types of code, so translating from guest syscalls to host syscalls is 
> relatively simple.
>
> However shared libraries tend to have much less cleanly defined 
> interfaces. They tend do share data structures, and be much more 
> closely linked. This makes adding the translation layer between the 
> two much more difficult, if not impossible. It generally requires 
> designing the interface with this in mind from the start, and in 
> general can't be retrofitted to existing libraries. Shared libraries 
> (aka dlls) share an address space with the main application, so tend 
> to be very hard to disentangle from each other.
>
> Paul
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>
--
When life hands you lemons, ask for a bottle of gin and tonic.





[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2545 bytes --]

Re: [Qemu-devel] Endian and userspace issues

[anarkhos]

At 8:11 PM -0800 1/4/05, John Davidorff Pell wrote:
>I think that part of what he is suggesting is that the code that is little endian be translated to Big endian before execution. This would make the running binary "native" in memory, and so could continue to be closely integrated with its linked libraries.

Yes, that was what I was suggesting. If this were done I don't even see the need for thunks at all.

Re: [Qemu-devel] Endian and userspace issues

[Karl Magdsick]

The difficulty comes when, for instance, you have a struct containing
a u_int16_t, followed by a int32_t[2], followed by a u_int8_t[2].  You
pass a pointer to this struct to an AES or TwoFish encryption
implementation that takes a void*.  Internally, this void* is treated
as a u_int32_t*.  If the struct were little endian, the most
significant byte of the u_int16_t would end up being in the middle of
the first 32-bit word as viewed by the encryption function.  The
trouble is that the necessary type casts in C/C++ don't follow through
to the machine code.  You can guess where the typecasts come, but it
becomes difficult.

Now, the way one would first think to implement x86 emulation on a
big-endian architecture would be to implement strait-forward access of
int8_ts and insert thunks for int16_ts, int32_ts, floats, doubles, and
long doubles.  In other words, the "normal" way to emulate a CPU on
top of a CPU of the oposite endianess will result in code optimized
for 8-bit memory accesses.

I imagine that the VirtualPC "big-endian x86" implementation really
isn't big-endian, but it instead optimizes the memory layout for
aligned 32-bit access.  I imagine the memory is viewed as an array of
alligned 32-bit big-endian words instead of an array of bytes. 
Loading (and storing) int32_ts and floats requires no endian changes
as long as the memory accesses are 32-bit aligned.  Loading (or
storing) an int8_t would require a simple thunk of XORing its address
with 0x3 and loading an aligned int16_t would require a simple thunk
of XORing its address with 0x2.  Doubles, long doubles, and unaligned
accesses require more complicated thunks.  However, aligned int32_ts
are the fastest accesses on 32-bit x86 CPUs, so performance tuned code
will use aligned int32_ts wherever possible.  Optimizing memory access
for aligned 32-bit access is likely a net positive gain.

If you're emulating an architecture that doesn't allow unaligned
memory accesses, then the gains are even greater.

Called native libraries would still need to be aware of the
non-standard memory layout, or else the emulatior will need to have
knowledge of the APIs and insert extra thunks in the function calls
between emulated and native code.

If you were emulating only user-space, I imagine you could insert
alternative implementations of calloc() and malloc() that set aside
some space for access accounting.  You could then see if the first
access to the allocated memory was an aligned 32-bit access and mark
the allocated buffer as optimized for 32-bit aligned access, otherwise
you would use "normal" emulation.  The accounting overhead and
complexity would likely make this "mixed endian memory layout"
emulation more trouble than it's worth.  With system emulation, you
could do something similar with accounting on a per-page or
per-kilobyte basis.


-Karl

On Tue, 4 Jan 2005 20:17:13 -0800, anarkhos@vfemail.net
<anarkhos@vfemail.net> wrote:
> At 8:11 PM -0800 1/4/05, John Davidorff Pell wrote:
> >I think that part of what he is suggesting is that the code that is little endian be translated to Big endian before execution. This would make the running binary "native" in memory, and so could continue to be closely integrated with its linked libraries.
> 
> Yes, that was what I was suggesting. If this were done I don't even see the need for thunks at all.
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>

Re: [Qemu-devel] Endian and userspace issues

[Gwenole Beauchesne]

On Wed, 5 Jan 2005, Karl Magdsick wrote:

> I imagine that the VirtualPC "big-endian x86" implementation really
> isn't big-endian,

Isn't Virtual PC switching the process to run in little endian mode? At 
the very least, you still have byte-reverse load/store instructions.

Re: [Qemu-devel] Endian and userspace issues

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

On 05.01.2005, at 11:34, Gwenole Beauchesne wrote:

> Isn't Virtual PC switching the process to run in little endian mode? At
> the very least, you still have byte-reverse load/store instructions.

So I thought. That's the reason why it didn't run on a G5 for quite
some time; other than the G3 and G4 the G5 isn't dual-endian.

Servus,
       Daniel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 478 bytes --]

Re: [Qemu-devel] Endian and userspace issues

[Magnus Damm]

On Wed, 5 Jan 2005 14:03:14 +0100, Daniel Egger <de@axiros.com> wrote:
> On 05.01.2005, at 11:34, Gwenole Beauchesne wrote:
> 
> > Isn't Virtual PC switching the process to run in little endian mode? At
> > the very least, you still have byte-reverse load/store instructions.
> 
> So I thought. That's the reason why it didn't run on a G5 for quite
> some time; other than the G3 and G4 the G5 isn't dual-endian.

Huh? 
But the byte-swapping 32-bit load and store instructions work just
fine on a G5, right?

/ magnus

Re: [Qemu-devel] Endian and userspace issues

[Daniel Egger]

[-- Attachment #1: Type: text/plain, Size: 283 bytes --]

On 05.01.2005, at 14:38, Magnus Damm wrote:

> Huh?
> But the byte-swapping 32-bit load and store instructions work just
> fine on a G5, right?

The G5 is in some way castrated regarding to endinaness. I don't
know offhand where exactly the limitations are...

Servus,
       Daniel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 478 bytes --]