Differences on Intel and Amd

Differences on Intel and Amd

[Alpár Török]

Hi all,

Sorry if this issues, or parts of this issue have been covered in
separate threads.

We  have an executable, of unknown origin, that is very likely to be
malicious.  We use KVM ( version 78) for sand-boxing the execution of
such software.
Each time the Virtual Machine is started from  a snapshot (with loadvm
), an executable is copied from a share and launched.

Now the problem. We have both AMD and Intel  machines. A snapshot
taken on Intel doesn't load on AMD and vice versa, so The snapshots
from which the VMs are
started are different. This executable, runs on the AMD machines, but
not on Intel. We concluded that the executable uses an undocumented
Windows API
function, and relies on a side-effect (a value placed in a register).
The value of this register differs from AMD to Intel. That is why it
shortly and silently terminates if ran on Intel. (by ran on Intel and
ran on AMD i mean of course a KVM VM on those platforms)

The questions are:
 Can a process within the VM find out the native processor type?
 Or can Windows XP find out the original processor type and behave differently?
 Does this behavior make sense to you?
 Is it possible that the difference is not due to hardware
differences, but because of different snapshots, and the events that
 occur before the snapshots, are different?

 We need to have consistent and repeatable results with thease
sand-boxed tests, that was what triggered the investigation in the
first place.


--
Alpar Torok

Re: Differences on Intel and Amd

[Avi Kivity]

On 09/21/2009 01:05 PM, Alpár Török  wrote:
> Hi all,
>
> Sorry if this issues, or parts of this issue have been covered in
> separate threads.
>
> We  have an executable, of unknown origin, that is very likely to be
> malicious.  We use KVM ( version 78) for sand-boxing the execution of
> such software.
> Each time the Virtual Machine is started from  a snapshot (with loadvm
> ), an executable is copied from a share and launched.
>
> Now the problem. We have both AMD and Intel  machines. A snapshot
> taken on Intel doesn't load on AMD and vice versa, so The snapshots
> from which the VMs are
> started are different. This executable, runs on the AMD machines, but
> not on Intel. We concluded that the executable uses an undocumented
> Windows API
> function, and relies on a side-effect (a value placed in a register).
> The value of this register differs from AMD to Intel. That is why it
> shortly and silently terminates if ran on Intel. (by ran on Intel and
> ran on AMD i mean of course a KVM VM on those platforms)
>    

On really new kvms this is no longer true, you can now load a snapshot 
saved on a processor from another vendor.

> The questions are:
>   Can a process within the VM find out the native processor type?
>    

Yes.  The vendor ID is not virtualized (you can override it with 
-cpu,vendor= to force virtualization).

>   Or can Windows XP find out the original processor type and behave differently?
>    

Yes, and it does.  Windows (and Linux) use different system call 
instructions depending on vendor.

>   Does this behavior make sense to you?
>    

Yes and no.  It makes sense since the x86 instruction set is not truly 
portable across vendors.  It doesn't since it should.

>   Is it possible that the difference is not due to hardware
> differences, but because of different snapshots, and the events that
>   occur before the snapshots, are different?
>
>    

It is due to hardware differences.

>   We need to have consistent and repeatable results with thease
> sand-boxed tests, that was what triggered the investigation in the
> first place.
>
>    

Try newer kvms.  We now emulate the differences (with some performance 
loss).

-- 
error compiling committee.c: too many arguments to function

Re: Differences on Intel and Amd

[Alpár Török]

Alpar Torok

[...]
>
> On really new kvms this is no longer true, you can now load a snapshot saved
> on a processor from another vendor.

I am really happy to hear that, another good thing about the latest
release is that loadvm from console works. Previously Win XP BSOD-ed.

[..]

In deed, this solves the problem. With kvm 88 ONLY if the -cpu
athlon,vendorId=AuthenticAMD option is present, the virtual cpu shows
up indeed as an AMD, and the executable runs as expected. In KVM 77
the option seems to have no effect.

 An interesting thing is that the sample that didn't run in a VM
hosted on Intel, runs fine, no matter what the host architecture is as
long as -cpu athlon,vendorId=AuthenticAMD OR  -cpu
core2duo,vendor=GenuineIntel is passed. This makes me think that by
default some processor parameters are different that they would be in
real hardware. It is thus possible that the malicious executable
performs VM detection (but fails if run on AMD host).

I will see how KVM behaves with this option when other executables are
tested, but i think that setting this option for a sand-box is
recommended (it might affect live migration, but i am not interested
in that).