Added VM Exit on RDTSC, trouble handling in userspace

Added VM Exit on RDTSC, trouble handling in userspace

[Kurt Kiefer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

In short, I have a need for trapping RDTSC with a VM Exit and this  
works, but I'm having trouble handling it in userspace. I have added  
the hooks I need (I only care about VMX right now), but a piece of the  
puzzle is missing and I don't know which. When I go back to userspace,  
it's triggering a different (faulty) execution vs. handling only in  
the kernel. Here's what I've done:


1. Added the CPU_BASED_RDTSC_EXITING flag to  
MSR_IA32_VMX_PROCBASED_CTLS in vmx.c:setup_vmcs_config()


2. Defined KVM_EXIT_RDTSC, and hooked into EXIT_REASON_RDTSC my  
handler for the exit:

static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu,
				      struct kvm_run *kvm_run) = {
// ...
   [EXIT_REASON_RDTSC]                   = handle_rdtsc,
// ...
}

static int handle_rdtsc(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
{
   u64 data;

   if (vmx_get_msr(vcpu, MSR_IA32_TIME_STAMP_COUNTER, &data)) {
     kvm_inject_gp(vcpu, 0);
     return 1;
   }

   vcpu->run->exit_reason = KVM_EXIT_RDTSC;
   vcpu->arch.regs[VCPU_REGS_RAX] = data & -1u;
   vcpu->arch.regs[VCPU_REGS_RDX] = (data >> 32) & -1u;

   skip_emulated_instruction(vcpu);

   // flag a need for userspace invervention
   // note: this works when we return 1 and we don't involve userspace
   return 0;
}


3. Handle KVM_EXIT_RDTSC in libkvm.c:kvm_run() :

     case KVM_EXIT_RDTSC:
       r = handle_rdtsc_usp(kvm, vcpu, env);
       break;

via a handler where I do _nothing_ :

static int handle_rdtsc_usp(kvm_context_t kvm, int vcpu, void *data)
{
   return 0;
}



All well and good, right? I can add print statements to my userspace  
handle_rtsc_usp() and see I get in there just fine. However, when I  
try to boot Linux, the following code is called over and over and  
over, and Linux will never load:

Breakpoint 4, 0x00000000c01103d3 in ?? ()
(gdb) x/10i $rip-10
0xc01103c9:	lea    0x0(%rdi,%riz,1),%edi
0xc01103d0:	push   %rbp
0xc01103d1:	mov    %esp,%ebp
0xc01103d3:	rdtsc
0xc01103d5:	pop    %rbp
0xc01103d6:	retq

If I only handle the exit in the kernel (by returning 1 from  
handle_rdtsc()), everything works and Linux will load! I counted the  
number of RDTSC exits before linux fully loads to be somewhere around  
200000. If I exit all the way to userspace (return 0 in my  
handle_rdtsc()) that count is infinitely surpassed in number of exits,  
wall time, and the value of RDTSC.

So is anything glaringly wrong with my modifications? Maybe there is  
there some extra state that needs to be restored on VM entry? Is there  
an interrupt flag that needs to be cleared? Maybe I need to do  
something with kvm_run.if_flag or  
kvm_run.ready_for_interrupt_injection? Please, I need help, I'm losing  
sleep over this!


Thanks,

Kurt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)

iEYEARECAAYFAkrVZvQACgkQYFGmU9mnI1FqvgCcC/+PswoXHQ5kVgv5tC6UadiA
KKgAoKrLgsYSJN0+1d0pox9vzsLHoQIc
=cQzR
-----END PGP SIGNATURE-----

Re: Added VM Exit on RDTSC, trouble handling in userspace

[Marcelo Tosatti]

On Tue, Oct 13, 2009 at 10:51:48PM -0700, Kurt Kiefer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> In short, I have a need for trapping RDTSC with a VM Exit and this  
> works, but I'm having trouble handling it in userspace. I have added the 
> hooks I need (I only care about VMX right now), but a piece of the  
> puzzle is missing and I don't know which. When I go back to userspace,  
> it's triggering a different (faulty) execution vs. handling only in the 
> kernel. Here's what I've done:
>
>
> 1. Added the CPU_BASED_RDTSC_EXITING flag to MSR_IA32_VMX_PROCBASED_CTLS 
> in vmx.c:setup_vmcs_config()
>
>
> 2. Defined KVM_EXIT_RDTSC, and hooked into EXIT_REASON_RDTSC my handler 
> for the exit:
>
> static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu,
> 				      struct kvm_run *kvm_run) = {
> // ...
>   [EXIT_REASON_RDTSC]                   = handle_rdtsc,
> // ...
> }
>
> static int handle_rdtsc(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
> {
>   u64 data;
>
>   if (vmx_get_msr(vcpu, MSR_IA32_TIME_STAMP_COUNTER, &data)) {
>     kvm_inject_gp(vcpu, 0);
>     return 1;
>   }
>
>   vcpu->run->exit_reason = KVM_EXIT_RDTSC;
>   vcpu->arch.regs[VCPU_REGS_RAX] = data & -1u;
>   vcpu->arch.regs[VCPU_REGS_RDX] = (data >> 32) & -1u;
>
>   skip_emulated_instruction(vcpu);
>
>   // flag a need for userspace invervention
>   // note: this works when we return 1 and we don't involve userspace
>   return 0;
> }
>
>
> 3. Handle KVM_EXIT_RDTSC in libkvm.c:kvm_run() :
>
>     case KVM_EXIT_RDTSC:
>       r = handle_rdtsc_usp(kvm, vcpu, env);
>       break;
>
> via a handler where I do _nothing_ :
>
> static int handle_rdtsc_usp(kvm_context_t kvm, int vcpu, void *data)
> {
>   return 0;
> }
>
>
>
> All well and good, right? I can add print statements to my userspace  
> handle_rtsc_usp() and see I get in there just fine. However, when I try 
> to boot Linux, the following code is called over and over and over, and 
> Linux will never load:
>
> Breakpoint 4, 0x00000000c01103d3 in ?? ()
> (gdb) x/10i $rip-10
> 0xc01103c9:	lea    0x0(%rdi,%riz,1),%edi
> 0xc01103d0:	push   %rbp
> 0xc01103d1:	mov    %esp,%ebp
> 0xc01103d3:	rdtsc
> 0xc01103d5:	pop    %rbp
> 0xc01103d6:	retq
>
> If I only handle the exit in the kernel (by returning 1 from  
> handle_rdtsc()), everything works and Linux will load! I counted the  
> number of RDTSC exits before linux fully loads to be somewhere around  
> 200000. If I exit all the way to userspace (return 0 in my  
> handle_rdtsc()) that count is infinitely surpassed in number of exits,  
> wall time, and the value of RDTSC.
>
> So is anything glaringly wrong with my modifications? Maybe there is  
> there some extra state that needs to be restored on VM entry? Is there  
> an interrupt flag that needs to be cleared? Maybe I need to do something 
> with kvm_run.if_flag or kvm_run.ready_for_interrupt_injection? Please, I 
> need help, I'm losing sleep over this!

Can't see anything wrong. Perhaps the userspace exit breaks a latency
assumption of that algorithm (can you translate it to function names?).

Try the tsc.flat test (from qemu-kvm.git):

x86_64-softmmu/qemu-system-x86_64 -chardev file,path=/tmp/log.txt,id=testlog \
-device testdev,chardev=testlog -kernel kvm/user/test/x86/tsc.flat