Difference between vcpu_load and kvm_sched_in ?

Difference between vcpu_load and kvm_sched_in ?

[Yacine]

Hi, I'm a student working on virtual machine introspection.

I'm trying to implement an application on top of KVM in which I need to trap
writes to CR3 (host with 8 cores and guest with one vcpu).

When I do this when handling a VM EXIT using:
vmcs_set_bits(CPU_BASED_VM_EXEC_CONTROL, CPU_BASED_CR3_LOAD_EXITING), it
works correctly and I can see the traps in my log file.

Now when I do the same thing after receiving a command from Qemu (command is
handled in kvm_vm_ioctl by calling a function I added to kvm_x86_ops
vmx_x86_ops) I get a vmwrite error. I found out that the problem is because
the logical processor on the host that is handling the ioctl command is not
the same that is running the VM and holding its state; so I must do the
vmwrite on the one executing the VM

To change the logical cpu executing the VM, I tried this:

vcpu_load; start cr3 trapping; vcpu_put

it worked correctly (in my logs I see that vcpu.cpu become equal to "cpu =
raw_smp_processor_id();") but the VM blocks for a lot of time due to mutex
in vcpu_load (up to serveral seconds and sometimes minutes !)

I replaced vcpu_load with kvm_sched_in, now everything works perfectly and
the VM doesn't block at all (logs here: http://pastebin.com/h5XNNMcb).

So, what I want to know is: what is the difference between vcpu_load and
kvm_sched_in ? both of this functions call kvm_arch_vcpu_loadbut the latter
one does it without doing a mutex

Is there a problem in using kvm_sched_in instead of vcpu_load for my use case ?

Re: Difference between vcpu_load and kvm_sched_in ?

[Paolo Bonzini]

On 20/10/2015 11:57, Yacine wrote:
> vcpu_load; start cr3 trapping; vcpu_put
> 
> it worked correctly (in my logs I see that vcpu.cpu become equal to "cpu =
> raw_smp_processor_id();") but the VM blocks for a lot of time due to mutex
> in vcpu_load (up to serveral seconds and sometimes minutes !)

Right, that's because while the CPU is running the mutex is taken.  If
the VCPU doesn't exit, the mutex is held.

> I replaced vcpu_load with kvm_sched_in, now everything works perfectly and
> the VM doesn't block at all (logs here: http://pastebin.com/h5XNNMcb).
> 
> So, what I want to know is: what is the difference between vcpu_load and
> kvm_sched_in ? both of this functions call kvm_arch_vcpu_loadbut the latter
> one does it without doing a mutex

kvm_sched_out and kvm_sched_in are part of KVM's preemption hooks.  The
hooks are registered only between vcpu_load and vcpu_put, therefore they
know that the mutex is taken.  The sequence will go like this:

    vcpu_load
    kvm_sched_out
    kvm_sched_in
    kvm_sched_out
    kvm_sched_in
    ...
    vcpu_put

and it will all happen with the mutex held.

> Is there a problem in using kvm_sched_in instead of vcpu_load for my use case ?

Yes, unfortunately it is a problem: you are loading the same VMCS on two
processors, which has undefined results.

To fix the problem, wrap the ioctl into a function and pass the function
to QEMU's "run_on_cpu" function.  It will send the ioctl from the right
thread, so that the kernel will not be holding the vcpu mutex.

Paolo

Re: Difference between vcpu_load and kvm_sched_in ?

[Wanpeng Li]

On 10/20/15 11:44 PM, Paolo Bonzini wrote:
>
> On 20/10/2015 11:57, Yacine wrote:
>> vcpu_load; start cr3 trapping; vcpu_put
>>
>> it worked correctly (in my logs I see that vcpu.cpu become equal to "cpu =
>> raw_smp_processor_id();") but the VM blocks for a lot of time due to mutex
>> in vcpu_load (up to serveral seconds and sometimes minutes !)
> Right, that's because while the CPU is running the mutex is taken.  If
> the VCPU doesn't exit, the mutex is held.
>
>> I replaced vcpu_load with kvm_sched_in, now everything works perfectly and
>> the VM doesn't block at all (logs here: http://pastebin.com/h5XNNMcb).
>>
>> So, what I want to know is: what is the difference between vcpu_load and
>> kvm_sched_in ? both of this functions call kvm_arch_vcpu_loadbut the latter
>> one does it without doing a mutex
> kvm_sched_out and kvm_sched_in are part of KVM's preemption hooks.  The
> hooks are registered only between vcpu_load and vcpu_put, therefore they
> know that the mutex is taken.  The sequence will go like this:
>
>      vcpu_load
>      kvm_sched_out
>      kvm_sched_in
>      kvm_sched_out
>      kvm_sched_in
>      ...
>      vcpu_put

If this should be:

vcpu_load
kvm_sched_in
kvm_sched_out
kvm_sched_in
kvm_sched_out
...
vcpu_put

Regards,
Wanpeng Li

Re: Difference between vcpu_load and kvm_sched_in ?

[Paolo Bonzini]

On 21/10/2015 00:57, Wanpeng Li wrote:
>> kvm_sched_out and kvm_sched_in are part of KVM's preemption hooks.  The
>> hooks are registered only between vcpu_load and vcpu_put, therefore they
>> know that the mutex is taken.  The sequence will go like this:
>>
>>      vcpu_load
>>      kvm_sched_out
>>      kvm_sched_in
>>      kvm_sched_out
>>      kvm_sched_in
>>      ...
>>      vcpu_put
> 
> If this should be:
> 
> vcpu_load
> kvm_sched_in
> kvm_sched_out
> kvm_sched_in
> kvm_sched_out
> ...
> vcpu_put

No, because vcpu_load is called while the thread is running.  Therefore,
the first preempt notifier call will be a sched_out notification, which
calls kvm_arch_vcpu_put.  Extending the picture above:

      vcpu_load        -> kvm_arch_vcpu_load
      kvm_sched_out    -> kvm_arch_vcpu_put
      kvm_sched_in     -> kvm_arch_vcpu_load
      kvm_sched_out    -> kvm_arch_vcpu_put
      kvm_sched_in     -> kvm_arch_vcpu_load
      ...
      kvm_sched_out    -> kvm_arch_vcpu_put
      kvm_sched_in     -> kvm_arch_vcpu_load
      vcpu_put         -> kvm_arch_vcpu_put

Thanks,

Paolo

Re: Difference between vcpu_load and kvm_sched_in ?

[Yacine HEBBAL]

Paolo Bonzini <pbonzini <at> redhat.com> writes:

> 
> 
> On 21/10/2015 00:57, Wanpeng Li wrote:
> >> kvm_sched_out and kvm_sched_in are part of KVM's preemption hooks.  The
> >> hooks are registered only between vcpu_load and vcpu_put, therefore they
> >> know that the mutex is taken.  The sequence will go like this:
> >>
> >>      vcpu_load
> >>      kvm_sched_out
> >>      kvm_sched_in
> >>      kvm_sched_out
> >>      kvm_sched_in
> >>      ...
> >>      vcpu_put
> > 
> > If this should be:
> > 
> > vcpu_load
> > kvm_sched_in
> > kvm_sched_out
> > kvm_sched_in
> > kvm_sched_out
> > ...
> > vcpu_put
> 
> No, because vcpu_load is called while the thread is running.  Therefore,
> the first preempt notifier call will be a sched_out notification, which
> calls kvm_arch_vcpu_put.  Extending the picture above:
> 
>       vcpu_load        -> kvm_arch_vcpu_load
>       kvm_sched_out    -> kvm_arch_vcpu_put
>       kvm_sched_in     -> kvm_arch_vcpu_load
>       kvm_sched_out    -> kvm_arch_vcpu_put
>       kvm_sched_in     -> kvm_arch_vcpu_load
>       ...
>       kvm_sched_out    -> kvm_arch_vcpu_put
>       kvm_sched_in     -> kvm_arch_vcpu_load
>       vcpu_put         -> kvm_arch_vcpu_put
> 
> Thanks,
> 
> Paolo
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Thanks for the explanation, it's very clear.
I tired that but I didn't succeed to send the ioctl from "run_on_cpu"
function, I didn't find how to set the right CPUStat
I've tried "current_cpu"

kvm_main.c:

// yacine.begin

static void do_vmi_start_kvm_ioctl(void *type) {
    printf("do_vmi_start_kvm_ioctl\n");
    kvm_vm_ioctl(kvm_state, type);
}

int vmi_start_kvm_ioctl(int type) { <- called from hmp.c
    printf("vmi_start_kvm_ioctl\n");
    run_on_cpu(current_cpu, do_vmi_start_kvm_ioctl, (void *) &type);
    return 0;
}
// yacine.end

This gives me a segmentation fault
Then I tired to replace current_cpu with ENV_GET_CPU(mon_get_cpu()), it
didn't work, I get nothing, no error but doesn't work
I tried also to pass mon->mon_cpu  through int vmi_start_kvm_ioctl(int type)
by adding a first parameter as CPUStat, i get compiler error "dereference
pointer to incomplete type"
I'm beginner to qemu and kvm code, can you please orient me to fix this
problem ? Thanks in advance

Yacine

Re: Difference between vcpu_load and kvm_sched_in ?

[Paolo Bonzini]

On 21/10/2015 12:17, Hebbal Yacine wrote:
> Thanks for the explanation, it's very clear.
> I tired that but I didn't succeed to send the ioctl from "run_on_cpu"
> function, I didn't find how to set the right CPUStat
> I've tried "current_cpu"

Current_cpu is always NULL outside the VCPU thread.

> 
> kvm_main.c:
> 
> // yacine.begin
> 
> static void do_vmi_start_kvm_ioctl(void *type) {
>     printf("do_vmi_start_kvm_ioctl\n");
>     kvm_vm_ioctl(kvm_state, type);

Are you sure you want a VM ioctl and not a VCPU ioctl?  Or perhaps a VM
ioctl to do generic processing, and a VCPU ioctl that is then sent to
all VCPUs?

If you use a VCPU ioctl, you can use CPU_FOREACH or a for loop to
iterate over all VCPUs.

Paolo

Re: Difference between vcpu_load and kvm_sched_in ?

[Yacine HEBBAL]

Paolo Bonzini <pbonzini <at> redhat.com> writes:

> 
> 
> On 21/10/2015 12:17, Hebbal Yacine wrote:
> > Thanks for the explanation, it's very clear.
> > I tired that but I didn't succeed to send the ioctl from "run_on_cpu"
> > function, I didn't find how to set the right CPUStat
> > I've tried "current_cpu"
> 
> Current_cpu is always NULL outside the VCPU thread.
> 
> > 
> > kvm_main.c:
> > 
> > // yacine.begin
> > 
> > static void do_vmi_start_kvm_ioctl(void *type) {
> >     printf("do_vmi_start_kvm_ioctl\n");
> >     kvm_vm_ioctl(kvm_state, type);


//yacine.begin
int hmp_vmi_op_result = 0;

static void do_vmi_kvm_ioctl(void *type_ioctl) {
    int* type = (int*) type_ioctl;
    hmp_vmi_op_result = kvm_vcpu_ioctl(current_cpu, *type);
    //hmp_vmi_start_result = kvm_vm_ioctl(kvm_state, *type);
}

int vmi_kvm_ioctl(int type) {
    CPUState* cpu;

    CPU_FOREACH(cpu) {
        run_on_cpu(cpu, do_vmi_kvm_ioctl, &type);
    }
    return hmp_vmi_op_result;
}
//yacine.end

Yes, it works perfectly this way even when running multiple VCPUs, thank you
a lot :)
In fact, I was using an old version of qemu (1.5.x), and it doesn't have
CPU_FOREACH, i searched a little for to replace it, but without any luck. So
I upgraded my working version and now everything is cool 

> Are you sure you want a VM ioctl and not a VCPU ioctl?  Or perhaps a VM
> ioctl to do generic processing, and a VCPU ioctl that is then sent to
> all VCPUs?
>If you use a VCPU ioctl, you can use CPU_FOREACH or a for loop to
> iterate over all VCPUs.

In fact, I get the same result when using vm_ioctl or vcpu_ioctl.
If I correctly understood you last paragraph, it is better to use vm_ioctl
to do generic processing that doesn't rely on a given VCPU and hence I won't
need to use "CPU_FOREACH, run_on_cpu and current_cpu".

Thanks again :)

> 
> Paolo
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Difference between vcpu_load and kvm_sched_in ?

[Wanpeng Li]

On 10/21/15 2:46 PM, Paolo Bonzini wrote:
>
> On 21/10/2015 00:57, Wanpeng Li wrote:
>>> kvm_sched_out and kvm_sched_in are part of KVM's preemption hooks.  The
>>> hooks are registered only between vcpu_load and vcpu_put, therefore they
>>> know that the mutex is taken.  The sequence will go like this:
>>>
>>>       vcpu_load
>>>       kvm_sched_out
>>>       kvm_sched_in
>>>       kvm_sched_out
>>>       kvm_sched_in
>>>       ...
>>>       vcpu_put
>> If this should be:
>>
>> vcpu_load
>> kvm_sched_in
>> kvm_sched_out
>> kvm_sched_in
>> kvm_sched_out
>> ...
>> vcpu_put
> No, because vcpu_load is called while the thread is running.  Therefore,
> the first preempt notifier call will be a sched_out notification, which
> calls kvm_arch_vcpu_put.  Extending the picture above:
>
>        vcpu_load        -> kvm_arch_vcpu_load
>        kvm_sched_out    -> kvm_arch_vcpu_put
>        kvm_sched_in     -> kvm_arch_vcpu_load
>        kvm_sched_out    -> kvm_arch_vcpu_put
>        kvm_sched_in     -> kvm_arch_vcpu_load
>        ...
>        kvm_sched_out    -> kvm_arch_vcpu_put
>        kvm_sched_in     -> kvm_arch_vcpu_load
>        vcpu_put         -> kvm_arch_vcpu_put

Got it, thanks. :-)

Regards,
Wanpeng Li

Re: Difference between vcpu_load and kvm_sched_in ?

[Paolo Bonzini]

On 21/10/2015 19:21, Yacine HEBBAL wrote:
> If I correctly understood you last paragraph, it is better to use vm_ioctl
> to do generic processing that doesn't rely on a given VCPU and hence I won't
> need to use "CPU_FOREACH, run_on_cpu and current_cpu".

Right.  On the other hand, you definitely want a vcpu_ioctl if you need
to call vcpu_load.

Thanks,

Paolo